[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 3.0.0 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-2.fc40 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 488875704 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f53f0-0x000f53ff] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5200 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1D87 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1C23 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001BE3 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1C97 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1D27 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1D5F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1c23-0xbffe1c96] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1c22] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1c97-0xbffe1d26] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1d27-0xbffe1d5e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1d5f-0xbffe1d86] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002372] x2apic enabled [ 0.003010] Switched APIC routing to physical x2apic. [ 0.004013] kvm-guest: setup PV IPIs [ 0.007414] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010013] pid_max: default: 32768 minimum: 301 [ 0.011141] LSM: Security Framework initializing [ 0.012000] Yama: becoming mindful. [ 0.012000] SELinux: Initializing. [ 0.012075] *** VALIDATE selinux *** [ 0.021101] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026589] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027165] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029124] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030128] *** VALIDATE tmpfs *** [ 0.032307] *** VALIDATE proc *** [ 0.034092] *** VALIDATE cgroup *** [ 0.035010] *** VALIDATE cgroup2 *** [ 0.036266] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037149] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039000] Spectre V2 : User space: Vulnerable [ 0.039000] Speculative Store Bypass: Vulnerable [ 0.041176] debug: unmapping init [mem 0xffffffff85059000-0xffffffff85060fff] [ 0.043000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043709] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044025] ... version: 2 [ 0.044983] ... bit width: 48 [ 0.045031] ... generic registers: 4 [ 0.046013] ... value mask: 0000ffffffffffff [ 0.047016] ... max period: 00007fffffffffff [ 0.048027] ... fixed-purpose events: 3 [ 0.049011] ... event mask: 000000070000000f [ 0.050300] rcu: Hierarchical SRCU implementation. [ 0.052423] smp: Bringing up secondary CPUs ... [ 0.053482] x86: Booting SMP configuration: [ 0.054016] .... node #0, CPUs: #1 #2 #3 [ 0.057560] smp: Brought up 1 node, 4 CPUs [ 0.059011] smpboot: Max logical packages: 1 [ 0.060024] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.216356] node 0 deferred pages initialised in 154ms [ 0.220140] devtmpfs: initialized [ 0.221225] x86/mm: Memory block size: 128MB [ 0.223219] gcov: version magic: 0x41383552 [ 0.226157] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.229086] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.231309] pinctrl core: initialized pinctrl subsystem [ 0.233169] [ 0.233580] ************************************************************* [ 0.235016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.237014] ** ** [ 0.240013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.242014] ** ** [ 0.244016] ** This means that this kernel is built to expose internal ** [ 0.246016] ** IOMMU data structures, which may compromise security on ** [ 0.249019] ** your system. ** [ 0.251011] ** ** [ 0.253016] ** If you see this message and you are not debugging the ** [ 0.255022] ** kernel, report this immediately to your vendor! ** [ 0.258020] ** ** [ 0.260012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.262014] ************************************************************* [ 0.264655] NET: Registered protocol family 16 [ 0.266358] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.270068] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.272063] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.275423] cpuidle: using governor menu [ 0.277513] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.280552] PCI: Using configuration type 1 for base access [ 0.282153] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.294097] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.296103] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.300025] cryptd: max_cpu_qlen set to 1000 [ 0.301232] ACPI: Added _OSI(Module Device) [ 0.303031] ACPI: Added _OSI(Processor Device) [ 0.304013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.305014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.309362] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.316316] ACPI: Interpreter enabled [ 0.317060] ACPI: PM: (supports S0 S3 S4 S5) [ 0.319017] ACPI: Using IOAPIC for interrupt routing [ 0.320119] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.323499] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.332283] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.334038] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.337020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.340089] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.345296] acpiphp: Slot [2] registered [ 0.346137] acpiphp: Slot [5] registered [ 0.348162] acpiphp: Slot [6] registered [ 0.349073] acpiphp: Slot [7] registered [ 0.351180] acpiphp: Slot [8] registered [ 0.352108] acpiphp: Slot [9] registered [ 0.354173] acpiphp: Slot [10] registered [ 0.355102] acpiphp: Slot [3] registered [ 0.357350] acpiphp: Slot [4] registered [ 0.359094] acpiphp: Slot [11] registered [ 0.360092] acpiphp: Slot [12] registered [ 0.361095] acpiphp: Slot [13] registered [ 0.363092] acpiphp: Slot [14] registered [ 0.364046] acpiphp: Slot [15] registered [ 0.365087] acpiphp: Slot [16] registered [ 0.366104] acpiphp: Slot [17] registered [ 0.368104] acpiphp: Slot [18] registered [ 0.370115] acpiphp: Slot [19] registered [ 0.371074] acpiphp: Slot [20] registered [ 0.372087] acpiphp: Slot [21] registered [ 0.373114] acpiphp: Slot [22] registered [ 0.375100] acpiphp: Slot [23] registered [ 0.376132] acpiphp: Slot [24] registered [ 0.378091] acpiphp: Slot [25] registered [ 0.379074] acpiphp: Slot [26] registered [ 0.380113] acpiphp: Slot [27] registered [ 0.382081] acpiphp: Slot [28] registered [ 0.383085] acpiphp: Slot [29] registered [ 0.384086] acpiphp: Slot [30] registered [ 0.385080] acpiphp: Slot [31] registered [ 0.387079] PCI host bridge to bus 0000:00 [ 0.389022] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.391029] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.393025] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.395025] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.397024] pci_bus 0000:00: root bus resource [mem 0x380000000000-0x38007fffffff window] [ 0.400024] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.401163] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.403938] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.407166] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.419019] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.424459] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.427034] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.429017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.431022] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.434520] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.436696] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.439045] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.442795] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.447014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.460959] pci 0000:00:02.0: reg 0x20: [mem 0x380000000000-0x380000003fff 64bit pref] [ 0.464901] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.471347] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.478021] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.485027] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.498037] pci 0000:00:05.0: reg 0x20: [mem 0x380000004000-0x380000007fff 64bit pref] [ 0.509514] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.516019] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.522016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.538032] pci 0000:00:06.0: reg 0x20: [mem 0x380000008000-0x38000000bfff 64bit pref] [ 0.546445] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.553023] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.558022] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.574024] pci 0000:00:07.0: reg 0x20: [mem 0x38000000c000-0x38000000ffff 64bit pref] [ 0.587000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.593020] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.598019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.612018] pci 0000:00:08.0: reg 0x20: [mem 0x380000010000-0x380000013fff 64bit pref] [ 0.622586] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.629020] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.634015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.649018] pci 0000:00:09.0: reg 0x20: [mem 0x380000014000-0x380000017fff 64bit pref] [ 0.660965] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.665022] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.670019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.681022] pci 0000:00:0a.0: reg 0x20: [mem 0x380000018000-0x38000001bfff 64bit pref] [ 0.694425] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.696388] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.698359] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.700382] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.701262] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.706037] iommu: Default domain type: Passthrough [ 0.709542] SCSI subsystem initialized [ 0.711170] ACPI: bus type USB registered [ 0.713097] usbcore: registered new interface driver usbfs [ 0.715109] usbcore: registered new interface driver hub [ 0.716054] usbcore: registered new device driver usb [ 0.717000] pps_core: LinuxPPS API ver. 1 registered [ 0.717000] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.720060] PTP clock support registered [ 0.722056] EDAC MC: Ver: 3.0.0 [ 0.724071] PCI: Using ACPI for IRQ routing [ 0.726280] NetLabel: Initializing [ 0.727015] NetLabel: domain hash size = 128 [ 0.729021] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.730096] NetLabel: unlabeled traffic allowed by default [ 0.732148] vgaarb: loaded [ 0.734303] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.736019] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.741378] clocksource: Switched to clocksource kvm-clock [ 0.849427] VFS: Disk quotas dquot_6.6.0 [ 0.851041] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.853338] *** VALIDATE ramfs *** [ 0.854441] *** VALIDATE hugetlbfs *** [ 0.855955] pnp: PnP ACPI init [ 0.858270] pnp: PnP ACPI: found 6 devices [ 0.877214] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.880217] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.882098] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.884130] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.886661] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.889029] pci_bus 0000:00: resource 8 [mem 0x380000000000-0x38007fffffff window] [ 0.892161] NET: Registered protocol family 2 [ 0.894923] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.899939] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.903579] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.910653] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.914445] TCP: Hash tables configured (established 65536 bind 65536) [ 0.917598] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.921136] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.923868] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.926867] NET: Registered protocol family 1 [ 0.930100] RPC: Registered named UNIX socket transport module. [ 0.932268] RPC: Registered udp transport module. [ 0.934129] RPC: Registered tcp transport module. [ 0.935733] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.938200] NET: Registered protocol family 44 [ 0.940123] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.942114] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.943416] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.945535] PCI: CLS 0 bytes, default 64 [ 0.947823] Unpacking initramfs... [ 2.370363] debug: unmapping init [mem 0xffff94cabcc54000-0xffff94cabffbffff] [ 2.374523] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.376782] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.380127] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.890930] Initialise system trusted keyrings [ 2.892632] Key type blacklist registered [ 2.894431] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.904444] zbud: loaded [ 2.907138] *** VALIDATE nfs *** [ 2.908190] *** VALIDATE nfs4 *** [ 2.909722] pstore: using deflate compression [ 2.912875] Platform Keyring initialized [ 3.003516] NET: Registered protocol family 38 [ 3.005281] Key type asymmetric registered [ 3.007083] Asymmetric key parser 'x509' registered [ 3.008982] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.011952] io scheduler mq-deadline registered [ 3.013875] io scheduler kyber registered [ 3.015670] io scheduler bfq registered [ 3.017557] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.020455] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.022859] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.025734] ACPI: Power Button [PWRF] [ 3.120827] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.211813] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.391400] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.492325] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.678900] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.707147] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.738839] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.747255] Non-volatile memory driver v1.3 [ 3.749427] Linux agpgart interface v0.103 [ 3.782415] virtio_blk virtio1: [vda] 133280 512-byte logical blocks (68.2 MB/65.1 MiB) [ 3.786054] vda: detected capacity change from 0 to 68239360 [ 3.800354] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.803353] vdb: detected capacity change from 0 to 1073741824 [ 3.818267] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.821231] vdc: detected capacity change from 0 to 2621440000 [ 3.843373] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.846480] vdd: detected capacity change from 0 to 2621440000 [ 3.861979] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.864767] vde: detected capacity change from 0 to 4294967296 [ 3.881915] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.884594] vdf: detected capacity change from 0 to 4294967296 [ 3.889816] libphy: Fixed MDIO Bus: probed [ 3.898542] usbcore: registered new interface driver usbserial_generic [ 3.901204] usbserial: USB Serial support registered for generic [ 3.903521] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.907965] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.909756] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.915620] mousedev: PS/2 mouse device common for all mice [ 3.918779] rtc_cmos 00:05: RTC can wake from S4 [ 3.922200] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.922362] rtc_cmos 00:05: registered as rtc0 [ 3.928058] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.928063] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.929570] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.931435] intel_pstate: CPU model not supported [ 3.941063] hid: raw HID events driver (C) Jiri Kosina [ 3.943086] usbcore: registered new interface driver usbhid [ 3.944904] usbhid: USB HID core driver [ 3.946566] drop_monitor: Initializing network drop monitor service [ 3.948932] Initializing XFRM netlink socket [ 3.950848] NET: Registered protocol family 10 [ 3.954049] Segment Routing with IPv6 [ 3.955624] NET: Registered protocol family 17 [ 3.957523] mpls_gso: MPLS GSO support [ 3.962742] RAS: Correctable Errors collector initialized. [ 3.965091] AVX version of gcm_enc/dec engaged. [ 3.966827] AES CTR mode by8 optimization enabled [ 4.036885] sched_clock: Marking stable (4036859496, 0)->(4968750264, -931890768) [ 4.039768] registered taskstats version 1 [ 4.041782] Loading compiled-in X.509 certificates [ 4.044139] zswap: loaded using pool lzo/zbud [ 4.064344] Key type big_key registered [ 4.075729] Key type encrypted registered [ 4.077196] ima: No TPM chip found, activating TPM-bypass! [ 4.079144] ima: Allocated hash algorithm: sha1 [ 4.080818] ima: No architecture policies found [ 4.082643] evm: Initialising EVM extended attributes: [ 4.084558] evm: security.selinux [ 4.085743] evm: security.ima [ 4.086777] evm: security.capability [ 4.088076] evm: HMAC attrs: 0x1 [ 4.090117] rtc_cmos 00:05: setting system clock to 2025-11-07 04:29:08 UTC (1762489748) [ 4.097928] debug: unmapping init [mem 0xffffffff86003000-0xffffffff861fffff] [ 4.101197] debug: unmapping init [mem 0xffffffff84d82000-0xffffffff85058fff] [ 4.109081] Write protecting the kernel read-only data: 28672k [ 4.112983] debug: unmapping init [mem 0xffffffff83403000-0xffffffff835fffff] [ 4.115891] debug: unmapping init [mem 0xffffffff83d14000-0xffffffff83dfffff] [ 4.148211] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.156891] systemd[1]: Detected virtualization kvm. [ 4.158973] systemd[1]: Detected architecture x86-64. [ 4.160721] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.193257] systemd[1]: No hostname configured. [ 4.195237] systemd[1]: Set hostname to . [ 4.197439] random: systemd: uninitialized urandom read (16 bytes read) [ 4.199721] systemd[1]: Initializing machine ID from random generator. [ 4.328233] random: systemd: uninitialized urandom read (16 bytes read) [ 4.331262] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 4.335904] random: systemd: uninitialized urandom read (16 bytes read) [ 4.338677] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 4.343374] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Reached target Timers. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... Starting Setup Virtual Console... Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Paths. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.880527] device-mapper: uevent: version 1.0.3 [ 4.882536] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.530431] virtio_net virtio0 ens2: renamed from eth0 [ 5.543014] random: fast init done [ 5.574238] scsi host0: ata_piix [ 5.618640] scsi host1: ata_piix [ 5.620188] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.622627] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.771774] dracut-initqueue[591]: RTNETLINK answers: File exists [ 10.602602] random: crng init done [ 10.604098] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.812557] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.871479] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.106035] SELinux: Disabled at runtime. [ 12.161657] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.170194] systemd[1]: Detected virtualization kvm. [ 12.172089] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.587962] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.591292] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.596261] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.600500] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.604091] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.612495] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.616231] systemd[1]: Listening on initctl Compatibility Named Pipe. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Kernel Debug File System... Activating swap /dev/disk/by-label/SWAP... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-getty.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ 12.669666] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Remount Root and Kernel File Systems... Mounting Huge Pages File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting POSIX Message Queue File System... [ OK ] Listening on Process Core Dump Socket. Starting Apply Kernel Variables... [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target Slices. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 12.994185] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.259251] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.288613] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.459285] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.468043] EDAC sbridge: Ver: 1.1.2 [ 14.992356] Key type dns_resolver registered [ 15.297944] NFS: Registering the id_resolver key type [ 15.300152] Key type id_resolver registered [ 15.301960] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg108-server login: [ 41.481015] hrtimer: interrupt took 4903513 ns [ 52.274212] libcfs: loading out-of-tree module taints kernel. [ 52.304076] Key type ._llcrypt registered [ 52.306157] Key type .llcrypt registered [ 52.422525] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_hostid [ 68.151531] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 69.258780] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 69.290652] alg: No test for adler32 (adler32-zlib) [ 70.591599] Lustre: Lustre: Build Version: 2.16.58_106_gac70ede [ 71.234681] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 72.935281] Key type lgssc registered [ 74.384600] Lustre: Echo OBD driver; http://www.lustre.org/ [ 91.802184] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 94.744092] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 104.077992] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 109.905861] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 115.589455] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 128.198954] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 139.082214] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 139.142533] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 139.177847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 140.427226] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 140.458624] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 140.541513] Lustre: lustre-MDT0000: new disk, initializing [ 140.611887] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 140.641225] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 144.333356] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 157.413521] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 157.468564] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 157.516839] Lustre: 6515:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 157.544630] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 157.548355] Lustre: Skipped 1 previous similar message [ 157.606868] Lustre: lustre-MDT0001: new disk, initializing [ 157.668505] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 157.697259] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 157.714959] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 161.386811] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 165.930955] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 174.414777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 174.478844] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 174.831088] Lustre: lustre-OST0000: new disk, initializing [ 174.833913] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 174.925560] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 176.497991] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 176.518408] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 176.660328] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 180.286217] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 195.236953] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 195.339857] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 195.573923] Lustre: lustre-OST0001: new disk, initializing [ 195.576653] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 195.666443] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 201.979842] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 204.378849] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 204.394658] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 204.481083] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 214.303941] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 220.727200] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 231.091734] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing check_logdir /tmp/testlogs/ [ 234.843527] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing yml_node [ 238.800460] Lustre: DEBUG MARKER: Client: 2.16.58.106 [ 240.912836] Lustre: DEBUG MARKER: MDS: 2.16.58.106 [ 242.642091] Lustre: DEBUG MARKER: OSS: 2.16.58.106 [ 243.882181] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Nov 6 23:33:06 EST 2025 [ 260.757860] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 262.392834] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 280.544452] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 280.548050] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 280.563473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 281.568321] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 281.570440] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 281.578146] Lustre: Skipped 2 previous similar messages [ 281.582528] Lustre: Skipped 3 previous similar messages [ 283.492392] LustreError: 12221:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 283.577837] Lustre: server umount lustre-MDT0000 complete [ 286.766133] LustreError: 6508:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762490031 with bad export cookie 16103560472160056763 [ 286.768271] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 286.781110] LustreError: 6508:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 291.808715] LustreError: 6523:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 291.812566] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 291.813265] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 291.824913] LustreError: 6523:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 291.846603] Lustre: Skipped 1 previous similar message [ 292.964139] LustreError: 12422:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 292.975563] LustreError: 12422:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 293.156862] Lustre: server umount lustre-MDT0001 complete [ 296.470586] LustreError: 12623:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 296.483269] LustreError: 12623:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 296.532352] Lustre: server umount lustre-OST0000 complete [ 299.673280] LustreError: 12824:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 299.685160] LustreError: 12824:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 299.844545] Lustre: server umount lustre-OST0001 complete [ 304.731588] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_hostid [ 312.815933] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 321.375906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 328.777598] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 335.735895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 342.057928] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 349.059536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 349.139191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 349.471763] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 349.525528] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 349.633783] Lustre: lustre-MDT0000: new disk, initializing [ 349.788974] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 349.826379] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 354.562626] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 366.675847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 366.745474] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 366.821168] Lustre: 15013:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 366.880714] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 366.883155] Lustre: Skipped 1 previous similar message [ 366.932844] Lustre: lustre-MDT0001: new disk, initializing [ 367.039064] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 367.091799] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 367.113448] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 371.205611] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 378.524298] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 382.143187] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 387.747290] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 387.832590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 388.019084] Lustre: lustre-OST0000: new disk, initializing [ 388.022257] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 388.106819] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 389.760929] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 389.771450] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 389.851175] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 392.771966] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 399.718288] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 403.870629] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 404.080914] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 407.687559] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 407.860944] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 410.592158] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 410.592957] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 410.598770] LustreError: Skipped 1 previous similar message [ 410.600629] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 410.609854] Lustre: Skipped 1 previous similar message [ 410.612025] Lustre: Skipped 1 previous similar message [ 415.669146] LustreError: 17951:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 415.674534] LustreError: 17951:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 415.713593] LustreError: 16743:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 415.801676] Lustre: server umount lustre-OST0000 complete [ 420.840395] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 420.851383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 420.855442] Lustre: Skipped 2 previous similar messages [ 425.072706] LustreError: 18153:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 425.089384] LustreError: 18153:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 425.267912] Lustre: server umount lustre-MDT0000 complete [ 425.956170] LustreError: 16893:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 425.966201] LustreError: 16893:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 427.861632] LustreError: 15004:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762490172 with bad export cookie 16103560472160058289 [ 427.866835] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 427.867211] LustreError: 15004:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 428.129930] Lustre: server umount lustre-MDT0001 complete [ 431.590876] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 23:36:14 (1762490174) [ 433.626545] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 435.108214] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 23:36:17 (1762490177) [ 439.854294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 440.186768] LustreError: 19023:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 440.251129] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 443.052971] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 445.410465] LustreError: 19023:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 450.025883] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 453.111331] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 457.377921] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 460.536532] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 465.993885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 470.819189] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 475.433514] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 496.099938] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 496.100336] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 496.107958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 496.115399] Lustre: Skipped 2 previous similar messages [ 496.123082] LustreError: Skipped 1 previous similar message [ 498.110460] LustreError: 21290:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 498.114682] LustreError: 21290:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 498.258278] Lustre: server umount lustre-OST0000 complete [ 507.434837] Lustre: server umount lustre-MDT0000 complete [ 510.218025] LustreError: 19002:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762490254 with bad export cookie 16103560472160059178 [ 510.225817] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 510.229243] LustreError: 19002:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 510.600190] Lustre: server umount lustre-MDT0001 complete [ 520.790693] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 522.789463] Key type lgssc unregistered [ 523.014647] LNet: 22223:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 523.019414] LNetError: 22223:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 523.036807] LNet: Removed LNI 192.168.201.108@tcp [ 523.583190] Key type .llcrypt unregistered [ 523.584947] Key type ._llcrypt unregistered [ 541.168676] Key type ._llcrypt registered [ 541.170426] Key type .llcrypt registered [ 541.258822] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 542.362401] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 542.388979] alg: No test for adler32 (adler32-zlib) [ 543.446350] Lustre: Lustre: Build Version: 2.16.58_106_gac70ede [ 543.640139] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 545.279475] Key type lgssc registered [ 546.217277] Lustre: Echo OBD driver; http://www.lustre.org/ [ 553.843312] loop: module loaded [ 553.929534] loop0: detected capacity change from 0 to 204800000 [ 554.112120] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 554.125584] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 554.485957] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 554.505899] blk_update_request: operation not supported error, dev loop0, sector 160240 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 554.552288] blk_update_request: operation not supported error, dev loop0, sector 160344 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 554.586038] blk_update_request: operation not supported error, dev loop0, sector 168552 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 554.655645] blk_update_request: operation not supported error, dev loop0, sector 173280 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 554.730423] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 560.843538] loop0: detected capacity change from 0 to 204800000 [ 561.001819] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 561.009329] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 561.023738] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 561.148491] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 565.805746] loop0: detected capacity change from 0 to 204800000 [ 570.587556] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 570.654194] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 570.672355] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 571.881366] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 571.921654] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 571.969725] Lustre: lustre-MDT0000: new disk, initializing [ 572.065567] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 572.082317] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 575.392305] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 584.363546] loop1: detected capacity change from 0 to 204800000 [ 590.441431] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 590.569591] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 590.982055] Lustre: lustre-OST07c6: new disk, initializing [ 590.984878] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 591.018441] Lustre: Skipped 1 previous similar message [ 591.119517] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 598.622465] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 600.689373] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 600.702887] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 600.766078] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 692.704589] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 692.722186] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 692.726235] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 697.833781] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 698.556835] LustreError: 26555:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 698.698816] Lustre: server umount lustre-OST07c6 complete [ 706.660166] LustreError: 27056:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 706.666926] LustreError: 27056:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 706.831950] Lustre: server umount lustre-MDT0000 complete [ 720.729112] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 723.278878] Key type lgssc unregistered [ 723.545807] LNet: 27889:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 723.559806] LNetError: 27889:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 724.589613] LNet: Removed LNI 192.168.201.108@tcp [ 725.393057] Key type .llcrypt unregistered [ 725.394213] Key type ._llcrypt unregistered [ 740.365331] Key type ._llcrypt registered [ 740.367187] Key type .llcrypt registered [ 740.439614] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 741.079779] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 741.089983] alg: No test for adler32 (adler32-zlib) [ 742.081369] Lustre: Lustre: Build Version: 2.16.58_106_gac70ede [ 742.270870] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 743.927174] Key type lgssc registered [ 744.746211] Lustre: Echo OBD driver; http://www.lustre.org/ [ 750.474612] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 750.494640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 751.859387] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 755.102586] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 762.337686] LustreError: 29119:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 762.355024] LustreError: 29119:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 763.542939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 763.911483] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 767.871755] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 772.733307] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 776.289951] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 781.337158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 781.600954] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 785.998129] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 790.840792] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 797.007307] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 805.096944] LustreError: 31338:0:(qsd_reint.c:618:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:2 until pending updates are flushed. [ 812.517467] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 812.518387] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 812.539029] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 812.544599] Lustre: Skipped 1 previous similar message [ 817.635494] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 818.192370] LustreError: 31592:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 818.372359] Lustre: server umount lustre-OST0000 complete [ 822.754063] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 822.756983] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 822.765802] Lustre: Skipped 2 previous similar messages [ 822.767827] Lustre: Skipped 1 previous similar message [ 827.874759] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 827.884201] Lustre: Skipped 2 previous similar messages [ 828.345399] LustreError: 31794:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 828.355564] LustreError: 31794:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 828.483288] Lustre: server umount lustre-MDT0000 complete [ 832.163435] LustreError: 29100:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762490576 with bad export cookie 623060342145914909 [ 832.175866] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 832.194236] LustreError: 29100:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 832.363461] LustreError: 31995:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 832.381811] LustreError: 31995:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 832.613490] Lustre: server umount lustre-MDT0001 complete [ 841.596984] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 844.027595] Key type lgssc unregistered [ 844.302489] LNet: 32533:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 844.308216] LNetError: 32533:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 844.320231] LNet: Removed LNI 192.168.201.108@tcp [ 845.016256] Key type .llcrypt unregistered [ 845.017903] Key type ._llcrypt unregistered [ 861.203329] Key type ._llcrypt registered [ 861.206297] Key type .llcrypt registered [ 861.297802] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 861.970743] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 862.040800] alg: No test for adler32 (adler32-zlib) [ 863.128251] Lustre: Lustre: Build Version: 2.16.58_106_gac70ede [ 863.386335] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 865.063168] Key type lgssc registered [ 866.374761] Lustre: Echo OBD driver; http://www.lustre.org/ [ 875.469748] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 875.522123] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 877.228455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 880.861726] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 882.658418] LustreError: 33785:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 882.673810] LustreError: 33785:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 886.752660] LustreError: 33785:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 888.152713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 888.398942] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 892.310723] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 896.801195] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 899.932573] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 905.005603] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 905.311506] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 910.360315] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 916.545542] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 921.720099] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 932.321367] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 932.324123] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 932.336926] Lustre: Skipped 1 previous similar message [ 932.346151] Lustre: Skipped 1 previous similar message [ 936.529075] LustreError: 36057:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 936.714840] Lustre: server umount lustre-OST0000 complete [ 942.563432] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 942.568884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 942.583591] Lustre: Skipped 1 previous similar message [ 942.589846] Lustre: Skipped 1 previous similar message [ 946.686282] LustreError: 36258:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 946.689384] LustreError: 36258:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 946.824749] Lustre: server umount lustre-MDT0000 complete [ 947.686978] LustreError: 34498:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 947.698658] LustreError: 34498:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 950.232791] LustreError: 36260:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762490694 with bad export cookie 1970728983042510188 [ 950.234430] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 950.252610] LustreError: 36260:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 950.466886] LustreError: 36462:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 950.470431] LustreError: 36462:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 950.679487] Lustre: server umount lustre-MDT0001 complete [ 961.221375] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 964.163696] Key type lgssc unregistered [ 964.483452] LNet: 36994:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 964.492225] LNetError: 36994:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 964.529093] LNet: Removed LNI 192.168.201.108@tcp [ 965.494589] Key type .llcrypt unregistered [ 965.500065] Key type ._llcrypt unregistered [ 988.825561] Key type ._llcrypt registered [ 988.835853] Key type .llcrypt registered [ 988.962744] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 989.941756] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 990.096420] alg: No test for adler32 (adler32-zlib) [ 991.167659] Lustre: Lustre: Build Version: 2.16.58_106_gac70ede [ 991.356146] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 993.015189] Key type lgssc registered [ 993.817259] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1001.445924] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1001.467984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1002.961935] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1006.645620] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 1013.221214] LustreError: 38392:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1013.240319] LustreError: 38392:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1014.746359] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1015.237977] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1019.086570] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 1024.345814] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1027.577889] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1033.814778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1034.121608] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1040.086469] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 1045.710474] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1048.503320] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1056.229075] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1056.240218] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1056.262881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1058.784359] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1058.785739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1058.809946] Lustre: Skipped 3 previous similar messages [ 1061.801207] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.8@tcp (stopping) [ 1061.936587] LustreError: 40638:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1062.039279] Lustre: server umount lustre-MDT0000 complete [ 1063.912067] LustreError: 38391:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1063.929675] LustreError: 38391:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1065.388789] LustreError: 38373:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762490809 with bad export cookie 10691146108708234614 [ 1065.397059] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1065.399026] LustreError: 38373:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1066.917097] LustreError: 38388:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1066.941341] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.8@tcp (stopping) [ 1069.024199] LustreError: 38386:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1069.024440] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1069.073232] Lustre: Skipped 1 previous similar message [ 1071.623903] LustreError: 40842:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1071.632928] LustreError: 40842:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1071.800568] Lustre: server umount lustre-MDT0001 complete [ 1092.278385] LustreError: 41066:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1092.290631] LustreError: 41066:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1092.499218] Lustre: server umount lustre-OST0000 complete [ 1109.014267] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 1111.778338] Key type lgssc unregistered [ 1112.085963] LNet: 41903:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1112.097966] LNetError: 41903:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1112.118896] LNet: Removed LNI 192.168.201.108@tcp [ 1112.875699] Key type .llcrypt unregistered [ 1112.877273] Key type ._llcrypt unregistered [ 1136.381174] Key type ._llcrypt registered [ 1136.386309] Key type .llcrypt registered [ 1136.525355] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 1137.137729] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1137.185835] alg: No test for adler32 (adler32-zlib) [ 1138.349303] Lustre: Lustre: Build Version: 2.16.58_106_gac70ede [ 1138.528619] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 1140.200596] Key type lgssc registered [ 1141.053430] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1148.239669] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1148.269053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1149.690918] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1153.205914] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 1160.166536] LustreError: 43301:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1160.182531] LustreError: 43301:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1160.628566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1161.011611] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1165.298309] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 1170.542687] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1174.181257] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1181.576939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1182.025705] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1187.406949] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 1192.709355] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1195.313297] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1203.665816] Lustre: lustre-OST0000: Not available for connect from 192.168.201.8@tcp (stopping) [ 1205.730542] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1205.733077] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1205.735062] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1205.751652] Lustre: Skipped 1 previous similar message [ 1207.873348] LustreError: 45546:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1208.104866] Lustre: server umount lustre-OST0000 complete [ 1226.210493] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1226.236092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1226.238623] Lustre: Skipped 1 previous similar message [ 1231.690522] LustreError: 45921:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1231.693697] LustreError: 45921:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1231.779402] Lustre: server umount lustre-MDT0000 complete [ 1232.358618] LustreError: 43301:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1232.367489] LustreError: 43301:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1235.100977] LustreError: 45922:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762490979 with bad export cookie 10583401867077258011 [ 1235.103305] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1235.121887] LustreError: 45922:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1241.332281] LustreError: 46124:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1241.334818] LustreError: 46124:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1241.623364] Lustre: server umount lustre-MDT0001 complete [ 1252.428253] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 1254.751258] Key type lgssc unregistered [ 1254.982797] LNet: 46658:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1255.001943] LNetError: 46658:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1255.023530] LNet: Removed LNI 192.168.201.108@tcp [ 1255.689533] Key type .llcrypt unregistered [ 1255.691340] Key type ._llcrypt unregistered [ 1274.297568] Key type ._llcrypt registered [ 1274.302473] Key type .llcrypt registered [ 1274.383476] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 1274.894792] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1274.945966] alg: No test for adler32 (adler32-zlib) [ 1276.007872] Lustre: Lustre: Build Version: 2.16.58_106_gac70ede [ 1276.250992] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 1277.921502] Key type lgssc registered [ 1278.972790] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1286.593540] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1286.644170] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1288.266761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1291.693754] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 1293.283361] LustreError: 48034:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1293.294438] LustreError: 48034:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1298.400157] LustreError: 48034:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1298.688393] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1298.961911] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1302.291362] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 1307.403542] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1310.477505] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1317.484829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1317.834700] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1322.963746] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 1327.984191] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1333.173761] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1338.433485] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1341.976602] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1354.011608] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2025-11-0623h51m36s [ 1355.763378] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1358.823466] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1358.828111] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1358.844785] Lustre: Skipped 2 previous similar messages [ 1358.855721] Lustre: Skipped 2 previous similar messages [ 1359.907955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1361.320098] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.8@tcp (stopping) [ 1361.330317] Lustre: Skipped 3 previous similar messages [ 1363.783936] LustreError: 50602:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1363.882995] Lustre: server umount lustre-MDT0000 complete [ 1365.985919] LustreError: 50064:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1366.013146] LustreError: 50064:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1366.792523] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1372.142965] LustreError: 50064:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1372.178120] LustreError: 50064:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1372.621763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1372.719411] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1372.955608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1373.036106] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1377.116850] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 1379.325934] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1381.113323] Lustre: DEBUG MARKER: Wait for df (36087) ... [ 1382.745487] Lustre: DEBUG MARKER: done [ 1388.513301] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1388.519808] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1388.549159] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1389.537133] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1392.562452] LustreError: 51659:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1392.577689] LustreError: 51659:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1392.713493] Lustre: server umount lustre-OST0000 complete [ 1399.776708] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1399.776892] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1399.789407] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1399.792820] Lustre: Skipped 3 previous similar messages [ 1402.029493] LustreError: 51861:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1402.031872] LustreError: 51861:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1402.174649] Lustre: server umount lustre-MDT0000 complete [ 1405.087918] LustreError: 48016:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762491149 with bad export cookie 9154687259783923770 [ 1405.089402] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1405.099932] LustreError: 48016:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1405.189363] LustreError: 52062:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1405.191615] LustreError: 52062:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1405.355438] Lustre: server umount lustre-MDT0001 complete [ 1413.422322] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 1416.041640] Key type lgssc unregistered [ 1416.317676] LNet: 52594:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1416.324642] LNetError: 52594:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1416.342286] LNet: Removed LNI 192.168.201.108@tcp [ 1417.190127] Key type .llcrypt unregistered [ 1417.196526] Key type ._llcrypt unregistered [ 1448.916806] Key type ._llcrypt registered [ 1448.920659] Key type .llcrypt registered [ 1449.023698] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_hostid [ 1463.635794] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 1464.897211] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1464.919886] alg: No test for adler32 (adler32-zlib) [ 1466.000223] Lustre: Lustre: Build Version: 2.16.58_106_gac70ede [ 1466.347918] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 1468.111181] Key type lgssc registered [ 1469.670867] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1479.351440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1485.473444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1491.073357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1497.051665] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1503.463172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1503.520856] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1503.531959] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1504.901122] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1504.960341] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1505.121822] Lustre: lustre-MDT0000: new disk, initializing [ 1505.269293] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1505.296612] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1509.355780] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 1518.561635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1518.640458] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1518.740463] Lustre: 56541:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1518.802536] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1518.805581] Lustre: Skipped 1 previous similar message [ 1518.866861] Lustre: lustre-MDT0001: new disk, initializing [ 1518.988769] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1519.038763] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1519.043816] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1522.340476] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 1529.479652] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1533.105216] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1539.776778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1539.843553] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1540.095348] Lustre: lustre-OST0000: new disk, initializing [ 1540.102761] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1540.199167] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1545.270817] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 1548.335642] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1548.343926] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1548.444064] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1551.779440] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1555.804066] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1555.975578] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1559.573777] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1559.747515] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1564.128428] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1564.130628] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1564.138038] LustreError: Skipped 1 previous similar message [ 1564.138849] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1564.152157] Lustre: Skipped 1 previous similar message [ 1569.252708] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1569.258153] Lustre: Skipped 2 previous similar messages [ 1574.373030] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1574.376794] Lustre: Skipped 1 previous similar message [ 1575.391876] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1575.441223] LustreError: 59482:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1575.576638] Lustre: server umount lustre-OST0000 complete [ 1579.491946] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1579.508710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1579.516577] Lustre: Skipped 1 previous similar message [ 1584.609308] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1584.613974] Lustre: Skipped 1 previous similar message [ 1584.684774] LustreError: 59684:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1584.687905] LustreError: 59684:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1584.809825] Lustre: server umount lustre-MDT0000 complete [ 1588.024590] LustreError: 56533:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762491332 with bad export cookie 8506341964584940144 [ 1588.030845] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1588.039768] LustreError: 56533:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1588.142596] LustreError: 59887:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1588.150240] LustreError: 59887:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1588.348623] Lustre: server umount lustre-MDT0001 complete [ 1596.003434] Lustre: DEBUG MARKER: == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 23:55:38 (1762491338) [ 1601.089361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1601.536890] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1605.245499] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 1606.630691] LustreError: 60559:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1606.652508] LustreError: 60559:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1610.720725] LustreError: 60559:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1612.765325] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1613.117950] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1616.312533] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 1620.850654] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1623.730802] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1629.475489] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1629.765386] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1634.504900] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 1639.000451] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1656.082780] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2025-11-0623h56m38s [ 1657.528782] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1664.216986] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1666.537645] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1666.631353] LustreError: 60554:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1686.990972] Lustre: lustre-MDT0000: Client 81e40e5f-9e57-44fa-bcae-a4984a1af4bf (at 192.168.201.8@tcp) reconnecting [ 1706.671178] LustreError: 60554:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1708.776540] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1711.325133] Lustre: DEBUG MARKER: done [ 1718.756390] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1718.766732] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1718.775140] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1718.793107] Lustre: Skipped 2 previous similar messages [ 1719.097188] LustreError: 63588:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1719.099955] LustreError: 63588:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1719.267665] Lustre: server umount lustre-OST0000 complete [ 1723.875631] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1723.884992] Lustre: Skipped 1 previous similar message [ 1728.626306] LustreError: 63788:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1728.631281] LustreError: 63788:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1728.734165] Lustre: server umount lustre-MDT0000 complete [ 1730.020176] LustreError: 63292:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1730.045744] LustreError: 63292:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1731.792140] LustreError: 60538:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762491476 with bad export cookie 8506341964584941019 [ 1731.794813] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1731.800859] LustreError: 60538:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1732.150739] Lustre: server umount lustre-MDT0001 complete [ 1741.346207] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 1743.752271] Key type lgssc unregistered [ 1743.989553] LNet: 64522:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1744.006551] LNetError: 64522:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1745.062876] LNet: Removed LNI 192.168.201.108@tcp [ 1745.806525] Key type .llcrypt unregistered [ 1745.808061] Key type ._llcrypt unregistered [ 1779.416376] Key type ._llcrypt registered [ 1779.417814] Key type .llcrypt registered [ 1779.559576] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_hostid [ 1793.010502] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 1793.801203] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1793.910241] alg: No test for adler32 (adler32-zlib) [ 1795.038327] Lustre: Lustre: Build Version: 2.16.58_106_gac70ede [ 1795.335631] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 1797.047126] Key type lgssc registered [ 1798.317552] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1807.129867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1813.463438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1818.385368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1823.813336] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1829.113813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1829.151696] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1829.162612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1830.409612] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1830.446571] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1830.509686] Lustre: lustre-MDT0000: new disk, initializing [ 1830.578540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1830.594235] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1833.174824] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 1843.203675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1843.276131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1843.349309] Lustre: 68485:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1843.411695] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1843.414756] Lustre: Skipped 1 previous similar message [ 1843.477277] Lustre: lustre-MDT0001: new disk, initializing [ 1843.574503] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1843.613089] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1843.627622] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1846.797815] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 1854.586876] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1858.598768] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1865.376129] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1865.461791] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1865.674754] Lustre: lustre-OST0000: new disk, initializing [ 1865.688591] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1865.780542] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1871.448662] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 1872.925641] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1872.933660] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1873.067451] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1879.010178] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1885.354743] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1885.725864] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1889.926267] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1890.134738] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1893.865121] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1893.867438] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1893.878395] LustreError: Skipped 1 previous similar message [ 1893.885889] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1893.925729] Lustre: Skipped 1 previous similar message [ 1893.980329] Lustre: Skipped 1 previous similar message [ 1898.180617] LustreError: 71427:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1898.463885] Lustre: server umount lustre-OST0000 complete [ 1904.099883] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1904.100389] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1904.107236] Lustre: Skipped 1 previous similar message [ 1904.112279] Lustre: Skipped 1 previous similar message [ 1909.219800] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1909.223278] Lustre: Skipped 1 previous similar message [ 1914.336887] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1914.344364] Lustre: Skipped 1 previous similar message [ 1916.383140] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1916.449294] LustreError: 71629:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1916.454550] LustreError: 71629:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1916.552165] Lustre: server umount lustre-MDT0000 complete [ 1919.786328] LustreError: 71428:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762491664 with bad export cookie 3049220349159820312 [ 1919.789719] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1919.801879] LustreError: 71428:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1919.954492] LustreError: 71845:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1919.958634] LustreError: 71845:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1920.145965] Lustre: server umount lustre-MDT0001 complete [ 1927.617415] Lustre: DEBUG MARKER: == conf-sanity test 36: df report consistency on OSTs with different block size ========================================================== 00:01:10 (1762491670) [ 1928.786123] Lustre: DEBUG MARKER: SKIP: conf-sanity test_36 remote OST [ 1930.189973] Lustre: DEBUG MARKER: == conf-sanity test 37: verify set tunables works for symlink device ========================================================== 00:01:13 (1762491673) [ 1937.570878] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 1940.374842] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1940.707268] LustreError: 72894:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1940.729640] LustreError: 72894:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1940.782786] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1941.589095] Lustre: Failing over lustre-MDT0000 [ 1941.593674] LustreError: 72923:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1941.599246] LustreError: 72923:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1941.604541] LustreError: 72923:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 1, retries 0, failed: rc = -5 [ 1941.703043] LustreError: 72974:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1941.706862] LustreError: 72974:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1941.852260] Lustre: server umount lustre-MDT0000 complete [ 1947.160597] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 00:01:30 (1762491690) [ 1952.865923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1953.228191] LustreError: 73544:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1953.278814] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1956.812242] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 1958.369334] LustreError: 73544:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1963.492915] LustreError: 73544:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1964.307815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1964.596025] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1968.874988] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 1974.085373] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1977.212678] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1983.481282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1983.767543] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1989.103573] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 1993.841848] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2003.246648] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2010.597489] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2010.615528] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2010.645943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2013.675338] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2015.581832] LustreError: 75891:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2015.591605] LustreError: 75891:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2015.654335] Lustre: server umount lustre-MDT0000 complete [ 2018.799028] LustreError: 73544:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2018.810950] LustreError: 73526:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762491763 with bad export cookie 3049220349159821467 [ 2018.812307] LustreError: 73544:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2018.824215] LustreError: 73526:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2018.824977] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2019.126405] Lustre: server umount lustre-MDT0001 complete [ 2021.955533] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2025.519267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2031.791515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2040.287343] Lustre: 66700:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1762491768/real 1762491768] req@000000009120769d x1848106328073600/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1762491784 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2040.308537] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2040.347511] Lustre: Skipped 1 previous similar message [ 2044.448078] LustreError: 66697:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000006fad6c56 x1848106328074368/t0(0) o250->MGC192.168.201.108@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2044.674623] LustreError: 75152:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2044.746443] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2044.836303] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2045.727187] Lustre: 66701:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1762491773/real 1762491773] req@00000000205f990c x1848106328073856/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1762491789 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2047.792594] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 2049.823155] Lustre: 66699:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1762491778/real 1762491778] req@0000000071295d71 x1848106328073984/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1762491794 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2054.944155] Lustre: 66699:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1762491783/real 1762491783] req@00000000a395e6a4 x1848106328074112/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1762491799 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2055.305670] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2059.510562] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 2060.782207] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2065.637461] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2068.973302] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2069.613206] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2076.128376] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2076.145561] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2076.179669] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2076.182177] Lustre: Skipped 3 previous similar messages [ 2082.064214] LustreError: 78355:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2082.074905] LustreError: 78355:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 2082.201072] Lustre: server umount lustre-MDT0000 complete [ 2084.322968] LustreError: 77381:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2084.348432] LustreError: 77381:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 2085.622169] LustreError: 74261:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762491830 with bad export cookie 3049220349159823371 [ 2085.631033] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2085.635094] LustreError: 74261:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 2085.998910] Lustre: server umount lustre-MDT0001 complete [ 2091.336890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2099.480135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2104.799883] Lustre: 66700:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1762491833/real 1762491833] req@000000000ab9b3e0 x1848106328098560/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1762491849 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2104.847692] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2104.873843] Lustre: Skipped 2 previous similar messages [ 2112.365213] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2112.369285] Lustre: Skipped 1 previous similar message [ 2112.424559] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2113.375371] Lustre: 66699:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1762491841/real 1762491841] req@00000000de975824 x1848106328099072/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1762491857 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2113.434238] Lustre: 66699:0:(client.c:2468:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 2115.901495] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 2120.680184] LustreError: 79413:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2120.690658] LustreError: 79413:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2123.041915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2126.848262] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 2128.876108] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2131.560061] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2134.055537] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2134.482717] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2138.593159] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2138.603862] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2138.611199] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2138.614331] Lustre: Skipped 3 previous similar messages [ 2144.421730] LustreError: 80813:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2144.438564] LustreError: 80813:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 2146.585268] Lustre: server umount lustre-MDT0000 complete [ 2150.071363] LustreError: 74261:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762491894 with bad export cookie 3049220349159824743 [ 2150.076337] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2150.083374] LustreError: 74261:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2150.395589] Lustre: server umount lustre-MDT0001 complete [ 2155.498755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2158.324637] Lustre: DEBUG MARKER: files compared the same [ 2170.451796] Lustre: server umount lustre-OST0000 complete [ 2184.771602] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 2186.851755] Key type lgssc unregistered [ 2187.056300] LNet: 82448:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2187.063126] LNetError: 82448:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2187.085959] LNet: Removed LNI 192.168.201.108@tcp [ 2187.622330] Key type .llcrypt unregistered [ 2187.624105] Key type ._llcrypt unregistered [ 2205.272264] Key type ._llcrypt registered [ 2205.274646] Key type .llcrypt registered [ 2205.375814] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 2205.923438] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2205.967330] alg: No test for adler32 (adler32-zlib) [ 2207.044430] Lustre: Lustre: Build Version: 2.16.58_106_gac70ede [ 2207.201716] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 2208.847163] Key type lgssc registered [ 2209.662503] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2216.219931] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2216.239876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2217.644814] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2220.953977] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug +malloc all [ 2223.073906] LustreError: 83825:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2223.088964] LustreError: 83825:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2228.206385] LustreError: 83825:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2228.340709] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2228.666252] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2231.802299] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug +malloc all [ 2236.227663] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2238.731631] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2243.717440] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2243.945348] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2249.006785] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug +malloc all [ 2253.695739] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2263.408934] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2273.762316] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2273.767260] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2273.773989] Lustre: Skipped 1 previous similar message [ 2277.524401] LustreError: 86100:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2277.636473] Lustre: server umount lustre-OST0000 complete [ 2284.001195] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2284.005059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2284.012092] Lustre: Skipped 2 previous similar messages [ 2286.743743] LustreError: 86301:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2286.747241] LustreError: 86301:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2286.807837] Lustre: server umount lustre-MDT0000 complete [ 2289.120268] LustreError: 83821:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2289.131338] LustreError: 83821:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2289.722366] LustreError: 83805:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762492034 with bad export cookie 9503742022928151117 [ 2289.723783] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2289.736595] LustreError: 83805:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2289.873493] LustreError: 86504:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2289.879474] LustreError: 86504:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2290.084744] Lustre: server umount lustre-MDT0001 complete [ 2297.925604] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 2300.043360] Key type lgssc unregistered [ 2300.262240] LNet: 87035:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2300.270199] LNetError: 87035:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2300.285227] LNet: Removed LNI 192.168.201.108@tcp [ 2300.892853] Key type .llcrypt unregistered [ 2300.895915] Key type ._llcrypt unregistered [ 2320.071193] Key type ._llcrypt registered [ 2320.072484] Key type .llcrypt registered [ 2320.198699] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 2321.211467] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2321.225381] alg: No test for adler32 (adler32-zlib) [ 2322.328837] Lustre: Lustre: Build Version: 2.16.58_106_gac70ede [ 2322.545474] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 2324.231169] Key type lgssc registered [ 2325.211856] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2331.793424] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2331.816808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2347.363371] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2352.479235] LustreError: 88384:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.108@tcp: failed processing log, type 4: rc = -110 [ 2379.168137] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2384.453810] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 2390.070743] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2397.852098] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2398.036930] Lustre: *** cfs_fail_loc=706, val=0*** [ 2398.464761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2398.534468] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2402.560635] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 2403.811384] LustreError: 88411:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2403.836235] LustreError: 88411:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2408.933641] LustreError: 88411:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2410.966266] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2411.438779] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2415.347889] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 2421.340441] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2425.336110] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2429.409096] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2429.412771] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2429.457741] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2429.922053] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2435.052149] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2435.055030] Lustre: Skipped 1 previous similar message [ 2440.168850] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2440.177147] Lustre: Skipped 1 previous similar message [ 2442.207122] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2442.271679] LustreError: 90708:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 2442.599849] Lustre: server umount lustre-OST0000 complete [ 2447.330561] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2447.338628] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2447.370555] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2447.373017] Lustre: Skipped 1 previous similar message [ 2450.410463] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2452.647741] LustreError: 90910:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 2452.657274] LustreError: 90910:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2452.769387] Lustre: server umount lustre-MDT0000 complete [ 2455.522635] LustreError: 89356:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2455.542533] LustreError: 89356:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2456.051472] LustreError: 88393:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762492200 with bad export cookie 5356271353525621643 [ 2456.055758] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2456.076265] LustreError: 88393:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2456.168844] LustreError: 91113:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2456.172077] LustreError: 91113:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2456.376428] Lustre: server umount lustre-MDT0001 complete [ 2467.970383] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 2470.886531] Key type lgssc unregistered [ 2471.197688] LNet: 91645:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2471.206832] LNetError: 91645:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2471.225533] LNet: Removed LNI 192.168.201.108@tcp [ 2472.040037] Key type .llcrypt unregistered [ 2472.041779] Key type ._llcrypt unregistered [ 2498.936548] Key type ._llcrypt registered [ 2498.938170] Key type .llcrypt registered [ 2499.010358] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 2500.233279] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2500.240086] alg: No test for adler32 (adler32-zlib) [ 2501.299847] Lustre: Lustre: Build Version: 2.16.58_106_gac70ede [ 2501.657696] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 2503.376045] Key type lgssc registered [ 2504.606107] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2513.168606] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2513.217855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2518.032578] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 2526.415579] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2526.847031] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2530.343509] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 2532.324099] LustreError: 93715:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2532.333591] LustreError: 93715:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2537.442555] LustreError: 93715:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2537.456436] LustreError: 93715:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2537.752729] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2538.072969] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2543.079129] LustreError: 93716:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2543.094434] LustreError: 93716:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2543.522693] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 2548.211801] LustreError: 93716:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2548.228769] LustreError: 93716:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2551.679990] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2551.700044] Lustre: MGS: Client a0835e84-d250-41dd-88ba-28a5e62b9091 (at 0@lo) reconnecting [ 2552.016653] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2552.022809] LustreError: 93726:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2552.091164] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2555.811766] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 2572.771239] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2572.789075] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2572.810192] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2573.795129] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2573.796504] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2573.810260] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2574.960820] LustreError: 95675:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 2575.103601] Lustre: server umount lustre-OST0000 complete [ 2578.912267] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2578.913861] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2578.921182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2578.923858] Lustre: Skipped 1 previous similar message [ 2578.932778] Lustre: Skipped 1 previous similar message [ 2584.032827] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2584.040360] Lustre: Skipped 2 previous similar messages [ 2584.501712] LustreError: 95877:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2584.503912] LustreError: 95877:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2584.517636] Lustre: server umount lustre-MDT0000 complete [ 2588.060732] LustreError: 96081:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2588.093640] LustreError: 96081:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2588.327147] Lustre: server umount lustre-MDT0001 complete [ 2592.183551] LustreError: 96284:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2592.185955] LustreError: 96284:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2592.237412] Lustre: server umount MGS complete [ 2603.446728] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 00:12:25 (1762492345) [ 2630.891381] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_hostid [ 2639.010752] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 2648.875368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2655.182457] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2661.262094] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2667.967469] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2675.049976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2675.124996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2679.339434] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 2688.951038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2689.026154] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2689.164510] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2689.214899] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2689.278575] Lustre: lustre-MDT0001: new disk, initializing [ 2689.355935] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2689.385047] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2689.393214] Lustre: 100802:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2689.402768] LustreError: 100802:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2692.540163] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 2701.311776] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2701.363948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2701.565334] Lustre: lustre-OST0000: new disk, initializing [ 2701.570128] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2701.640193] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2706.261172] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 2712.614179] Lustre: 102103:0:(client.c:1627:after_reply()) @@@ resending request on EINPROGRESS req@000000001184e8d3 x1848107069381632/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1762492473 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2713.339551] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2719.800262] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2719.815811] Lustre: MGS: Client fb919edb-d6e1-439f-a8a0-b7997a9ba700 (at 0@lo) reconnecting [ 2719.855577] Lustre: 100086:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2719.873143] LustreError: 100086:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 2719.913508] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2720.025665] Lustre: lustre-MDT0000: new disk, initializing [ 2720.147853] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2720.189732] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2721.258491] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2721.273122] Lustre: 102514:0:(client.c:1627:after_reply()) @@@ resending request on EINPROGRESS req@000000002d577057 x1848107069387520/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1762492481 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2724.869313] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 2730.413591] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2730.434116] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2730.485344] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2755.554724] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2755.556829] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2755.589061] Lustre: Skipped 1 previous similar message [ 2760.772148] LustreError: 103045:0:(obd_class.h:479:obd_check_dev()) Device 11 not setup [ 2760.775149] LustreError: 103045:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 2760.991687] Lustre: server umount lustre-OST0000 complete [ 2765.796686] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2765.806502] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2765.806779] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2765.818834] Lustre: Skipped 4 previous similar messages [ 2770.869429] LustreError: 103249:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 2770.890593] LustreError: 103249:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2770.963173] LustreError: 102944:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2770.964854] Lustre: server umount lustre-MDT0000 complete [ 2789.343449] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2789.513957] LustreError: 103450:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 2789.547691] LustreError: 103450:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2789.932953] Lustre: server umount lustre-MDT0001 complete [ 2794.825266] Lustre: server umount MGS complete [ 2809.310365] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 00:15:51 (1762492551) [ 2826.082838] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 2828.667635] Key type lgssc unregistered [ 2828.958946] LNet: 104990:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2828.963202] LNetError: 104990:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2828.980846] LNet: Removed LNI 192.168.201.108@tcp [ 2829.721731] Key type .llcrypt unregistered [ 2829.724741] Key type ._llcrypt unregistered [ 2846.062308] Key type ._llcrypt registered [ 2846.064709] Key type .llcrypt registered [ 2846.183009] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 2847.079743] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2847.189073] alg: No test for adler32 (adler32-zlib) [ 2848.191247] Lustre: Lustre: Build Version: 2.16.58_106_gac70ede [ 2848.447365] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 2850.143168] Key type lgssc registered [ 2851.013798] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2857.543344] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2857.570239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2859.015450] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2862.469970] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 2869.216096] LustreError: 106191:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2869.225454] LustreError: 106191:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2869.463697] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2869.863769] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2873.540200] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 2878.638700] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2882.018336] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2888.243147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2888.506266] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2893.505514] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 2897.707765] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2903.863095] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2915.296431] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2915.300389] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2915.309485] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2919.393512] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2919.396944] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2919.401633] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2920.817841] LustreError: 108457:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2920.946775] Lustre: server umount lustre-OST0000 complete [ 2924.531026] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2924.538300] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2924.541516] Lustre: Skipped 1 previous similar message [ 2929.632277] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2930.459954] LustreError: 108659:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2930.462341] LustreError: 108659:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2930.520224] Lustre: server umount lustre-MDT0000 complete [ 2934.277164] LustreError: 106171:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762492678 with bad export cookie 10106395041488344135 [ 2934.283372] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2934.303183] LustreError: 106171:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2934.529295] LustreError: 108860:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2934.548871] LustreError: 108860:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2934.852852] Lustre: server umount lustre-MDT0001 complete [ 2945.278843] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 2947.589630] Key type lgssc unregistered [ 2947.853657] LNet: 109393:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2947.863776] LNetError: 109393:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2947.879301] LNet: Removed LNI 192.168.201.108@tcp [ 2948.543112] Key type .llcrypt unregistered [ 2948.544815] Key type ._llcrypt unregistered [ 2962.831245] Key type ._llcrypt registered [ 2962.833143] Key type .llcrypt registered [ 2962.932502] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 2964.192560] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2964.296386] alg: No test for adler32 (adler32-zlib) [ 2965.422708] Lustre: Lustre: Build Version: 2.16.58_106_gac70ede [ 2965.789647] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 2967.551269] Key type lgssc registered [ 2968.505376] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2973.787442] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2973.789390] LustreError: 110458:0:(tgt_mount.c:2435:server_fill_super()) cfs_race id 716 sleeping [ 2973.793684] Lustre: Skipped 1 previous similar message [ 2973.801416] LustreError: 110455:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 waking [ 2973.805275] LustreError: 110458:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 awake: rc=4991 [ 2973.810782] LustreError: 110458:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2973.823843] LustreError: 110458:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2973.827550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2975.219529] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2982.441427] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2982.777828] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2986.999881] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 2992.748727] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing lsmod [ 2995.875041] LustreError: 111615:0:(tgt_mount.c:2435:server_fill_super()) cfs_race id 716 sleeping [ 2996.050112] LustreError: 111635:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 waking [ 2996.055452] LustreError: 111615:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 awake: rc=4829 [ 2996.060868] LustreError: 111615:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2996.075717] LustreError: 111615:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2996.077404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2996.437384] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3001.828539] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3001.831521] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3001.850750] Lustre: Skipped 1 previous similar message [ 3005.210846] LustreError: 111824:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3005.366733] Lustre: server umount lustre-MDT0000 complete [ 3006.946155] LustreError: 110494:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3006.967868] LustreError: 110494:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3007.970030] LustreError: 110497:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3008.008059] LustreError: 110497:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3009.418896] LustreError: 110476:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762492753 with bad export cookie 2132809658665796583 [ 3009.425031] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3009.436527] LustreError: 110476:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3010.143294] LustreError: 110930:0:(client.c:1370:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000009b9dfaf7 x1848107555901696/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3010.184710] LustreError: 112028:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3010.187773] LustreError: 112028:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3010.383507] Lustre: server umount lustre-MDT0001 complete [ 3024.584252] LustreError: 112231:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3024.589156] LustreError: 112231:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3024.799522] Lustre: server umount lustre-OST0000 complete [ 3032.427947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3032.909625] LustreError: 112708:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3033.005494] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3037.105733] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 3038.179265] LustreError: 112708:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3043.298153] LustreError: 112708:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3045.463203] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3045.830937] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3050.275587] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 3055.314793] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3058.668280] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3064.672716] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3064.997653] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3070.699790] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 3075.499683] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3078.333910] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3086.820839] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3086.832108] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3086.851548] Lustre: Skipped 1 previous similar message [ 3086.856220] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3086.863173] Lustre: Skipped 2 previous similar messages [ 3088.872183] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3088.877209] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3088.888289] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3088.907190] Lustre: Skipped 1 previous similar message [ 3091.387181] LustreError: 114954:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3091.390292] LustreError: 114954:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3091.604729] Lustre: server umount lustre-OST0000 complete [ 3097.056236] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3097.059215] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3097.082309] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3101.789644] LustreError: 115156:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3101.794127] LustreError: 115156:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3101.875857] Lustre: server umount lustre-MDT0000 complete [ 3104.224946] LustreError: 112708:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3104.242371] LustreError: 112708:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3105.944154] LustreError: 112687:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762492850 with bad export cookie 2132809658665797297 [ 3105.959538] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3106.228559] Lustre: server umount lustre-MDT0001 complete [ 3116.890505] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 3119.724567] Key type lgssc unregistered [ 3120.041499] LNet: 115889:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3120.054862] LNetError: 115889:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3120.074689] LNet: Removed LNI 192.168.201.108@tcp [ 3120.941922] Key type .llcrypt unregistered [ 3120.946804] Key type ._llcrypt unregistered [ 3147.410300] Key type ._llcrypt registered [ 3147.412701] Key type .llcrypt registered [ 3147.592861] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 3148.991241] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3149.405301] alg: No test for adler32 (adler32-zlib) [ 3150.502742] Lustre: Lustre: Build Version: 2.16.58_106_gac70ede [ 3150.794601] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 3152.527221] Key type lgssc registered [ 3154.056399] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3162.462875] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3162.478860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3163.896987] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3167.559651] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 3174.378845] LustreError: 117286:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3174.430753] LustreError: 117286:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3176.844137] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3177.258829] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3181.553341] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 3187.219613] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3190.948710] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3198.125418] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3198.522636] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3204.193131] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 3208.771252] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3209.929186] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3216.254549] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3219.418098] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3224.051302] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3224.053150] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3224.056298] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3224.056404] Lustre: Skipped 1 previous similar message [ 3225.225131] LustreError: 119631:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3225.446606] Lustre: server umount lustre-OST0000 complete [ 3233.760082] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3233.763192] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3233.771175] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3233.773622] Lustre: Skipped 1 previous similar message [ 3235.451101] LustreError: 119837:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3235.468356] LustreError: 119837:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3235.531715] Lustre: server umount lustre-MDT0000 complete [ 3239.402600] LustreError: 118655:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3239.441807] LustreError: 118655:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3239.609241] LustreError: 117266:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762492984 with bad export cookie 17879995033175458608 [ 3239.624486] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3239.627206] LustreError: 117266:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3239.824931] LustreError: 120039:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3239.829182] LustreError: 120039:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3240.254849] Lustre: server umount lustre-MDT0001 complete [ 3253.336268] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 3256.587581] Key type lgssc unregistered [ 3256.862779] LNet: 120570:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3256.867551] LNetError: 120570:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3256.883679] LNet: Removed LNI 192.168.201.108@tcp [ 3257.937629] Key type .llcrypt unregistered [ 3257.939967] Key type ._llcrypt unregistered [ 3276.440620] Key type ._llcrypt registered [ 3276.442169] Key type .llcrypt registered [ 3276.550314] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 3277.410028] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3277.497713] alg: No test for adler32 (adler32-zlib) [ 3278.510388] Lustre: Lustre: Build Version: 2.16.58_106_gac70ede [ 3278.809281] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 3280.487160] Key type lgssc registered [ 3281.677412] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3289.130834] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3289.148660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3290.544300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3293.713392] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 3299.817170] LustreError: 121772:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3299.848474] LustreError: 121772:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3301.783875] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3302.086564] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3305.188781] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 3312.422789] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3316.168567] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3322.709088] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3322.954417] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3329.802913] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 3335.233949] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3335.878872] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3345.381877] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3345.385168] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3345.389795] Lustre: Skipped 1 previous similar message [ 3345.394201] Lustre: Skipped 1 previous similar message [ 3349.439437] LustreError: 124021:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3349.559255] Lustre: server umount lustre-OST0000 complete [ 3353.568584] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3353.579180] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3353.611519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3355.617423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3355.618698] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3359.529471] LustreError: 124224:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3359.534140] LustreError: 124224:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3359.582473] Lustre: server umount lustre-MDT0000 complete [ 3360.737615] LustreError: 121768:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3360.753207] LustreError: 121768:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3363.367871] LustreError: 121754:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762493107 with bad export cookie 2505905085980810329 [ 3363.374457] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3363.384381] LustreError: 121754:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3363.548584] LustreError: 124425:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3363.557786] LustreError: 124425:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3363.759628] Lustre: server umount lustre-MDT0001 complete [ 3373.561141] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 3376.717184] Key type lgssc unregistered [ 3377.082218] LNet: 124957:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3377.088923] LNetError: 124957:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3378.154582] LNet: Removed LNI 192.168.201.108@tcp [ 3379.258322] Key type .llcrypt unregistered [ 3379.263444] Key type ._llcrypt unregistered [ 3401.643752] Key type ._llcrypt registered [ 3401.646900] Key type .llcrypt registered [ 3401.737091] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 3402.556571] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3402.602253] alg: No test for adler32 (adler32-zlib) [ 3403.629721] Lustre: Lustre: Build Version: 2.16.58_106_gac70ede [ 3403.815592] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 3405.497115] Key type lgssc registered [ 3406.386272] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3413.132517] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3413.179458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3414.606369] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3418.247516] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 3423.729784] LustreError: 126356:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3423.744666] LustreError: 126356:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3425.993771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3426.318233] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3429.334903] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 3434.577860] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3438.242976] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3444.266179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3444.721170] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3451.376665] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 3458.231832] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3459.083463] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3466.797752] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3470.546880] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3470.551984] Lustre: Skipped 2 previous similar messages [ 3474.883855] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3474.891376] Lustre: Skipped 2 previous similar messages [ 3476.542373] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3476.599831] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3482.153637] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.8@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3482.192707] Lustre: Skipped 2 previous similar messages [ 3485.501613] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.8@tcp 192.168.0.[2,10]@tcp [ 3490.773285] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.8/24@tcp in log lustre-MDT0000 [ 3490.778153] Lustre: Skipped 2 previous similar messages [ 3496.014019] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.8/24@tcp [ 3496.017913] Lustre: Skipped 1 previous similar message [ 3502.048461] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3502.054627] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3502.071207] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3504.103457] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3504.106322] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3504.125102] LustreError: Skipped 1 previous similar message [ 3504.161591] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3507.579106] LustreError: 129907:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3507.755041] Lustre: server umount lustre-OST0000 complete [ 3513.313758] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3513.317857] LustreError: Skipped 4 previous similar messages [ 3513.319989] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3513.327617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3513.330243] Lustre: Skipped 2 previous similar messages [ 3515.364147] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3515.366126] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3515.383176] Lustre: Skipped 1 previous similar message [ 3520.483442] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3520.494273] Lustre: Skipped 1 previous similar message [ 3525.090709] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3525.194402] LustreError: 130109:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3525.198575] LustreError: 130109:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3525.268178] Lustre: server umount lustre-MDT0000 complete [ 3525.601269] LustreError: 126353:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3525.622738] LustreError: 126353:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3528.583851] LustreError: 126336:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762493272 with bad export cookie 7209743254869002511 [ 3528.585157] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3528.597122] LustreError: 126336:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3528.742049] LustreError: 130311:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3528.744976] LustreError: 130311:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3529.039801] Lustre: server umount lustre-MDT0001 complete [ 3539.037713] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 3541.396608] Key type lgssc unregistered [ 3541.612954] LNet: 130841:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3541.621048] LNetError: 130841:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3541.632777] LNet: Removed LNI 192.168.201.108@tcp [ 3542.345775] Key type .llcrypt unregistered [ 3542.347968] Key type ._llcrypt unregistered [ 3566.911424] Key type ._llcrypt registered [ 3566.913254] Key type .llcrypt registered [ 3567.049697] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 3567.739190] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3567.891762] alg: No test for adler32 (adler32-zlib) [ 3568.979413] Lustre: Lustre: Build Version: 2.16.58_106_gac70ede [ 3569.162469] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 3570.848175] Key type lgssc registered [ 3571.968472] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3578.627343] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3578.645241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3579.976445] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3580.005369] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3580.029361] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.8/24@tcp [ 3583.419991] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 3589.993211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3590.111761] LustreError: 132364:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3590.120747] LustreError: 132364:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3590.237516] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3590.264561] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.8/24@tcp [ 3590.317412] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3593.250571] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 3598.142919] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3601.417626] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3607.131884] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3607.436757] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3612.378431] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 3616.880547] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3624.894404] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3636.724735] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3636.728408] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3636.739460] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3640.288783] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3640.290984] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3640.316325] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3640.360937] Lustre: Skipped 1 previous similar message [ 3641.918098] LustreError: 134692:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3642.137211] Lustre: server umount lustre-OST0000 complete [ 3646.433705] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3646.442250] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3646.466329] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3650.533313] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3650.535078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3655.650019] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3655.661516] Lustre: Skipped 2 previous similar messages [ 3660.255392] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3660.380761] LustreError: 134894:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3660.387172] LustreError: 134894:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3660.493174] Lustre: server umount lustre-MDT0000 complete [ 3663.922642] LustreError: 132343:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762493408 with bad export cookie 13348163244443014198 [ 3663.926165] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3663.966882] LustreError: 132343:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3664.066709] LustreError: 135096:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3664.078623] LustreError: 135096:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3664.308565] Lustre: server umount lustre-MDT0001 complete [ 3673.442690] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 3675.799750] Key type lgssc unregistered [ 3676.025259] LNet: 135628:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3676.041133] LNetError: 135628:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3676.062425] LNet: Removed LNI 192.168.201.108@tcp [ 3676.699314] Key type .llcrypt unregistered [ 3676.700629] Key type ._llcrypt unregistered [ 3702.373345] Key type ._llcrypt registered [ 3702.374962] Key type .llcrypt registered [ 3702.450807] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_hostid [ 3715.450653] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 3716.123543] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3716.178738] alg: No test for adler32 (adler32-zlib) [ 3717.147518] Lustre: Lustre: Build Version: 2.16.58_106_gac70ede [ 3717.332852] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 3718.985634] Key type lgssc registered [ 3720.039527] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3726.920942] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3733.268424] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3739.848111] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3745.983884] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3752.894218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3752.945674] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3752.961071] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3754.145306] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3754.180849] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3754.227863] Lustre: lustre-MDT0000: new disk, initializing [ 3754.299843] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3754.320955] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3757.933737] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 3769.771875] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3769.831616] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3769.911294] Lustre: 139002:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3769.995250] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3769.999393] Lustre: Skipped 1 previous similar message [ 3770.125334] Lustre: lustre-MDT0001: new disk, initializing [ 3770.264387] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3770.288605] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3770.302571] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3775.382671] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 3783.952814] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3787.601666] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3794.767776] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3794.892352] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3795.155729] Lustre: lustre-OST0000: new disk, initializing [ 3795.163537] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3795.273774] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3801.353135] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 3802.706660] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3802.722675] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3802.832038] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3808.658100] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3813.802156] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3814.058966] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3819.154612] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3819.413579] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3823.584509] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3823.595249] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3823.597104] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3823.636337] Lustre: Skipped 1 previous similar message [ 3827.533459] LustreError: 141943:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3827.798898] Lustre: server umount lustre-OST0000 complete [ 3833.825428] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3833.836619] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3833.842197] Lustre: Skipped 2 previous similar messages [ 3838.294574] LustreError: 142145:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3838.298796] LustreError: 142145:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3838.460708] Lustre: server umount lustre-MDT0000 complete [ 3842.266659] LustreError: 138993:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762493586 with bad export cookie 3994280823527474830 [ 3842.271380] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3842.296291] LustreError: 138993:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3842.411316] LustreError: 142347:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3842.417605] LustreError: 142347:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3842.693205] Lustre: server umount lustre-MDT0001 complete [ 3851.944335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3852.365255] LustreError: 142824:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3852.397460] LustreError: 142824:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3852.474564] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3856.583391] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 3857.888394] LustreError: 142824:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3861.989543] LustreError: 142824:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3864.189814] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3864.526848] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3867.768525] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 3872.215922] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3875.146844] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3881.195113] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3881.532182] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3886.809711] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 3892.182799] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3896.299209] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3897.592988] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 3901.414257] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3901.614167] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3911.091951] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3911.184564] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3911.462985] Lustre: lustre-OST0001: new disk, initializing [ 3911.473109] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3911.487381] Lustre: lustre-OST0001: Not available for connect from 192.168.201.8@tcp (not set up) [ 3911.573779] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3917.201259] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 3919.442476] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3919.458368] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3919.553285] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3924.694436] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3924.921515] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3928.184238] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3928.373854] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3931.567599] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9916863fa800.ost_server_uuid 50 [ 3932.972316] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9916863fa800.ost_server_uuid in FULL state after 0 sec [ 3943.181141] LustreError: 147080:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 3943.197658] LustreError: 147080:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3943.254671] Lustre: server umount lustre-OST0001 complete [ 3945.446595] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3945.452884] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3945.468756] LustreError: 144420:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3945.495462] Lustre: Skipped 2 previous similar messages [ 3948.000733] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3948.007867] LustreError: Skipped 1 previous similar message [ 3948.016042] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3948.033290] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3950.050759] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3950.567529] LustreError: 144422:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3950.582193] LustreError: 144422:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3952.386169] LustreError: 147282:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3952.388966] LustreError: 147282:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3952.558470] Lustre: server umount lustre-OST0000 complete [ 3956.704790] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3956.711505] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3956.720893] Lustre: Skipped 1 previous similar message [ 3956.741591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3956.744365] Lustre: Skipped 2 previous similar messages [ 3962.310484] LustreError: 147485:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 3962.317631] LustreError: 147485:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3962.446286] Lustre: server umount lustre-MDT0000 complete [ 3965.926434] LustreError: 143887:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3965.945509] LustreError: 143887:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3966.096616] LustreError: 146994:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762493710 with bad export cookie 3994280823527475712 [ 3966.101839] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3966.151765] LustreError: 146994:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3966.498921] Lustre: server umount lustre-MDT0001 complete [ 3985.000105] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 3988.001255] Key type lgssc unregistered [ 3988.348912] LNet: 148675:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3988.365784] LNetError: 148675:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3989.418752] LNet: Removed LNI 192.168.201.108@tcp [ 3990.435957] Key type .llcrypt unregistered [ 3990.442507] Key type ._llcrypt unregistered [ 4023.722290] Key type ._llcrypt registered [ 4023.729531] Key type .llcrypt registered [ 4023.815377] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_hostid [ 4036.903357] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 4038.090281] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4038.117307] alg: No test for adler32 (adler32-zlib) [ 4039.211827] Lustre: Lustre: Build Version: 2.16.58_106_gac70ede [ 4039.505226] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 4041.239726] Key type lgssc registered [ 4042.226546] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4050.294352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4055.826863] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4061.364792] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4068.084905] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4075.501390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4075.593935] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4075.633649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4076.988743] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4077.045983] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4077.236135] Lustre: lustre-MDT0000: new disk, initializing [ 4077.445238] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4077.490262] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4082.444876] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 4092.833723] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4092.887956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4092.953662] Lustre: 152629:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4093.014516] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4093.020688] Lustre: Skipped 1 previous similar message [ 4093.096162] Lustre: lustre-MDT0001: new disk, initializing [ 4093.197595] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4093.250588] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4093.264669] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4096.804599] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 4104.074752] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4107.347897] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4112.728287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4112.781275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4113.001258] Lustre: lustre-OST0000: new disk, initializing [ 4113.021916] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4113.110747] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4118.433786] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 4121.756123] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4121.791562] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4122.018208] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4127.032418] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4130.776370] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4130.956032] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4135.671798] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4135.897901] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4142.566078] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4142.567253] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4142.615184] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4147.696311] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4147.706650] Lustre: Skipped 1 previous similar message [ 4152.287980] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4152.336080] LustreError: 155568:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4152.529129] Lustre: server umount lustre-OST0000 complete [ 4157.922695] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4157.957560] Lustre: Skipped 1 previous similar message [ 4157.982098] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4157.992456] Lustre: Skipped 2 previous similar messages [ 4162.788918] LustreError: 155771:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4162.797938] LustreError: 155771:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4162.857680] Lustre: server umount lustre-MDT0000 complete [ 4166.782706] LustreError: 152622:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762493911 with bad export cookie 9219704375407624856 [ 4166.786633] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4166.796908] LustreError: 152622:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4166.956327] LustreError: 155972:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4166.963932] LustreError: 155972:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4167.436384] Lustre: server umount lustre-MDT0001 complete [ 4176.961616] Lustre: DEBUG MARKER: == conf-sanity test 47a: server restart does not lose lru_resize ========================================================== 00:38:39 (1762493919) [ 4191.840028] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_hostid [ 4200.567640] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 4209.636586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4216.107330] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4221.859936] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4227.450398] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4232.961426] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4233.006629] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4233.146822] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4233.171764] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4233.224567] Lustre: lustre-MDT0000: new disk, initializing [ 4233.290569] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4233.311235] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4236.277646] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 4245.194277] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4245.249046] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4245.305763] Lustre: 158975:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4245.344782] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4245.348847] Lustre: Skipped 1 previous similar message [ 4245.402572] Lustre: lustre-MDT0001: new disk, initializing [ 4245.480161] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4245.516958] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4245.541534] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4248.828245] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 4256.600749] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4260.390234] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4266.378313] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4266.454690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4266.639597] Lustre: lustre-OST0000: new disk, initializing [ 4266.642675] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4266.727452] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4272.051436] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 4274.229444] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4274.246055] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4274.345382] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4278.482071] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4282.236799] Lustre: Failing over lustre-OST0000 [ 4282.279494] LustreError: 161415:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4282.282908] LustreError: 161415:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4282.435943] Lustre: server umount lustre-OST0000 complete [ 4299.785734] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4299.998122] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4300.003316] Lustre: Skipped 1 previous similar message [ 4300.010341] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4300.022409] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4300.764202] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4301.023206] Lustre: 150841:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1762494029/real 1762494029] req@000000000ba6bb36 x1848108681075456/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1762494045 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4301.043049] Lustre: 150841:0:(client.c:2468:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 4305.446578] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4305.446720] Lustre: lustre-OST0000: Recovery over after 0:05, of 3 clients 3 recovered and 0 were evicted. [ 4305.814139] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 4306.399302] Lustre: 150840:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1762494034/real 1762494034] req@000000009b9dfaf7 x1848108681076736/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1762494050 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4306.414649] Lustre: 150840:0:(client.c:2468:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4310.141235] Lustre: Failing over lustre-MDT0000 [ 4310.280747] LustreError: 162395:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4310.309688] LustreError: 162395:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4310.450127] Lustre: server umount lustre-MDT0000 complete [ 4310.505048] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4310.508896] LustreError: 158982:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4310.512636] Lustre: Skipped 3 previous similar messages [ 4310.526915] LustreError: 158982:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 4311.519168] Lustre: 150842:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1762494039/real 1762494039] req@000000003b5cc523 x1848108681077888/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1762494055 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4311.537656] Lustre: 150842:0:(client.c:2468:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4315.618631] LustreError: 158980:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4315.630950] LustreError: 158980:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4320.740871] LustreError: 161328:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4320.758593] LustreError: 161328:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4325.857129] LustreError: 158986:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4325.885408] LustreError: 158986:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4328.505709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4328.783297] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4329.049794] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4332.165614] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 4332.466236] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4334.054081] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4334.062307] Lustre: Skipped 1 previous similar message [ 4334.078841] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 4334.108908] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4339.174241] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4339.178123] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4339.183537] Lustre: Skipped 1 previous similar message [ 4339.189164] Lustre: Skipped 1 previous similar message [ 4344.123236] LustreError: 163283:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4344.137363] LustreError: 163283:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4344.295984] Lustre: server umount lustre-OST0000 complete [ 4349.408438] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4349.412976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4349.435976] Lustre: Skipped 1 previous similar message [ 4353.482743] Lustre: server umount lustre-MDT0000 complete [ 4354.533592] LustreError: 160107:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4354.566719] LustreError: 160107:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4357.035847] LustreError: 158967:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762494101 with bad export cookie 9219704375407626970 [ 4357.044259] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4357.062736] LustreError: 158967:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4357.602504] Lustre: server umount lustre-MDT0001 complete [ 4367.625390] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 4370.666245] Key type lgssc unregistered [ 4370.974764] LNet: 164218:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4370.992987] LNetError: 164218:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4371.019351] LNet: Removed LNI 192.168.201.108@tcp [ 4371.926280] Key type .llcrypt unregistered [ 4371.928078] Key type ._llcrypt unregistered [ 4414.012920] Key type ._llcrypt registered [ 4414.015467] Key type .llcrypt registered [ 4414.173311] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 4414.959934] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4414.972118] alg: No test for adler32 (adler32-zlib) [ 4415.974273] Lustre: Lustre: Build Version: 2.16.58_106_gac70ede [ 4416.239784] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 4417.943165] Key type lgssc registered [ 4418.965391] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4425.744171] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4425.762685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4427.224865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4431.060796] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 4437.476012] LustreError: 165720:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4437.483414] LustreError: 165720:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4438.187793] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4441.379730] LustreError: 165749:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4441.386273] LustreError: 165749:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4441.391731] LustreError: 165749:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 14, retries 0, failed: rc = -5 [ 4441.478533] LustreError: 166272:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4441.617066] Lustre: server umount lustre-MDT0000 complete [ 4454.722663] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_hostid [ 4463.367953] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 4472.100490] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4478.173144] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4483.594711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4489.265290] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4495.661598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4495.710680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4495.998196] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4496.045756] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4496.129809] Lustre: lustre-MDT0000: new disk, initializing [ 4496.235531] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4496.259344] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4500.314323] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 4510.773912] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4510.826550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4510.876454] Lustre: 168915:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4510.915735] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4510.918352] Lustre: Skipped 1 previous similar message [ 4510.970348] Lustre: lustre-MDT0001: new disk, initializing [ 4511.073287] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4511.117770] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4511.130333] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4514.817572] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 4522.338153] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4526.378984] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4533.066249] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4533.123498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4533.577406] Lustre: lustre-OST0000: new disk, initializing [ 4533.592885] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4533.724694] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4540.147967] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 4543.548994] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4543.557542] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4543.656294] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4545.842459] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4550.026399] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4551.262867] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 00:44:54 (1762494294) [ 4782.560641] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4782.568734] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4782.590188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4784.611177] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4784.612109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4784.633879] Lustre: Skipped 1 previous similar message [ 4784.639537] Lustre: Skipped 1 previous similar message [ 4788.558658] LustreError: 172442:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4788.573506] LustreError: 172442:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4788.720780] Lustre: server umount lustre-MDT0000 complete [ 4789.736604] LustreError: 168920:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4789.749307] LustreError: 168920:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4792.859950] LustreError: 168908:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762494537 with bad export cookie 3323160193185137661 [ 4792.864715] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4794.854992] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4794.860234] LustreError: 168920:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4794.875965] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4797.923108] LustreError: 170899:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4797.923207] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4799.124636] LustreError: 172646:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4799.129797] LustreError: 172646:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4799.417439] Lustre: server umount lustre-MDT0001 complete [ 4809.776064] LustreError: 172848:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4809.782114] LustreError: 172848:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4809.920530] Lustre: server umount lustre-OST0000 complete [ 4818.275978] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_hostid [ 4829.673024] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 4840.000994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4847.696121] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4854.396980] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4861.105435] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4867.578843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4867.696613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4867.932733] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4867.976569] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4868.032172] Lustre: lustre-MDT0000: new disk, initializing [ 4868.130452] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4868.171161] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4871.501252] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 4881.534700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4881.596205] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4881.687262] Lustre: 175192:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4881.747521] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4881.750607] Lustre: Skipped 1 previous similar message [ 4881.798454] Lustre: lustre-MDT0001: new disk, initializing [ 4881.889521] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4881.937699] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4881.957673] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4885.905840] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 4892.123506] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4896.222629] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4903.536895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4903.645405] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4903.934738] Lustre: lustre-OST0000: new disk, initializing [ 4903.952561] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4904.080201] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4909.848391] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 4910.651564] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4910.663464] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4910.799512] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4917.095388] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4920.537621] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4920.673509] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4924.449481] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4924.698829] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4926.434556] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4926.435249] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4926.438701] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4926.439677] Lustre: Skipped 1 previous similar message [ 4926.442626] LustreError: Skipped 1 previous similar message [ 4932.409140] LustreError: 178129:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4932.419072] LustreError: 178129:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4932.566922] Lustre: server umount lustre-OST0000 complete [ 4936.676621] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4936.704240] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4936.712430] Lustre: Skipped 4 previous similar messages [ 4941.876494] LustreError: 178331:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4941.894361] LustreError: 178331:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4941.969837] Lustre: server umount lustre-MDT0000 complete [ 4945.267765] LustreError: 175182:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762494689 with bad export cookie 3323160193185296491 [ 4945.273514] LustreError: 175182:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4945.277651] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4945.777798] Lustre: server umount lustre-MDT0001 complete [ 4954.738139] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 00:51:37 (1762494697) [ 4969.652680] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_hostid [ 4977.610435] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 4985.526980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4991.233960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4996.093345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5001.403870] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5008.502769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5008.582046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5008.871720] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5008.901612] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5008.975514] Lustre: lustre-MDT0000: new disk, initializing [ 5009.088409] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5009.109115] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5013.094382] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 5022.091604] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5022.135804] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5022.282814] Lustre: 181524:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5022.493547] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5025.459154] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 5032.603748] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5036.921615] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5044.065751] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5044.136164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5044.376712] Lustre: lustre-OST0000: new disk, initializing [ 5044.383799] Lustre: Skipped 1 previous similar message [ 5044.387538] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5044.400992] Lustre: Skipped 2 previous similar messages [ 5050.294290] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 5050.441097] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5050.455068] Lustre: Skipped 1 previous similar message [ 5050.465371] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5050.552636] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5057.434707] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5066.211489] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5066.213945] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5066.231075] LustreError: Skipped 1 previous similar message [ 5066.231965] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5066.236241] Lustre: Skipped 2 previous similar messages [ 5066.269278] Lustre: Skipped 3 previous similar messages [ 5070.598756] LustreError: 184071:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5070.605982] LustreError: 184071:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 5070.969634] Lustre: server umount lustre-OST0000 complete [ 5081.083123] Lustre: server umount lustre-MDT0000 complete [ 5081.569118] LustreError: 181537:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5081.576398] LustreError: 181537:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5084.529361] LustreError: 181517:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762494828 with bad export cookie 3323160193185297373 [ 5084.533818] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5084.588625] LustreError: 181517:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5085.008195] Lustre: server umount lustre-MDT0001 complete [ 5093.567903] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 00:53:56 (1762494836) [ 5107.194349] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_hostid [ 5115.049119] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 5121.925633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5128.347414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5134.183443] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5140.870257] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5146.498644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5146.552648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5146.756968] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5146.786950] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5146.843296] Lustre: lustre-MDT0000: new disk, initializing [ 5146.910659] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5146.914146] Lustre: Skipped 2 previous similar messages [ 5146.927044] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5150.169493] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 5159.511024] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5159.575316] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5159.696247] Lustre: 187465:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5159.919774] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5162.979636] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 5170.656393] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5174.596829] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5181.039254] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5181.097569] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5186.766853] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 5187.154287] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5193.550917] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5200.628662] LustreError: 190015:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5200.632865] LustreError: 190015:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 5200.765576] Lustre: server umount lustre-OST0000 complete [ 5205.987324] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5205.994126] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5205.999665] Lustre: Skipped 2 previous similar messages [ 5206.005915] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5206.008651] Lustre: Skipped 2 previous similar messages [ 5213.769375] Lustre: server umount lustre-MDT0000 complete [ 5217.510428] LustreError: 187457:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762494961 with bad export cookie 3323160193185298458 [ 5217.518442] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5217.529354] LustreError: 187457:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5227.164973] Lustre: server umount lustre-MDT0001 complete [ 5235.669948] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 5237.851833] Key type lgssc unregistered [ 5238.080190] LNet: 190950:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5238.089600] LNetError: 190950:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5238.099097] LNet: Removed LNI 192.168.201.108@tcp [ 5238.711802] Key type .llcrypt unregistered [ 5238.713517] Key type ._llcrypt unregistered [ 5257.883680] Key type ._llcrypt registered [ 5257.886113] Key type .llcrypt registered [ 5258.017866] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 5258.928343] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5258.960157] alg: No test for adler32 (adler32-zlib) [ 5259.970176] Lustre: Lustre: Build Version: 2.16.58_106_gac70ede [ 5260.199249] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 5261.850552] Key type lgssc registered [ 5262.884436] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5268.956912] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5268.970293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5270.266655] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5272.859754] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 5279.715969] LustreError: 192327:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5279.744459] LustreError: 192327:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5279.828766] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5280.079726] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5283.395714] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 5288.103614] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5291.488926] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5297.035474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5297.301993] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5301.541921] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 5306.160040] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5315.771902] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5336.547991] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5336.548554] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5336.557897] LustreError: Skipped 1 previous similar message [ 5336.558735] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5336.573216] Lustre: Skipped 1 previous similar message [ 5339.189218] LustreError: 194597:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5339.368316] Lustre: server umount lustre-OST0000 complete [ 5346.785246] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5346.788734] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5346.798513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5346.817386] Lustre: Skipped 1 previous similar message [ 5346.825386] Lustre: Skipped 2 previous similar messages [ 5351.906605] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5351.919529] Lustre: Skipped 1 previous similar message [ 5357.025107] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5357.026492] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 4. Is it stuck? [ 5357.028785] Lustre: Skipped 1 previous similar message [ 5357.102798] LustreError: 194798:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5357.107547] LustreError: 194798:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5357.173789] Lustre: server umount lustre-MDT0000 complete [ 5360.213052] LustreError: 192307:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762495104 with bad export cookie 12087003000759204543 [ 5360.218797] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5360.226555] LustreError: 192307:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5360.346958] LustreError: 195003:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5360.352214] LustreError: 195003:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5360.640151] Lustre: server umount lustre-MDT0001 complete [ 5369.730675] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 5372.525210] Key type lgssc unregistered [ 5372.771153] LNet: 195534:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5372.778864] LNetError: 195534:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5372.788160] LNet: Removed LNI 192.168.201.108@tcp [ 5373.569344] Key type .llcrypt unregistered [ 5373.572175] Key type ._llcrypt unregistered [ 5392.229920] Key type ._llcrypt registered [ 5392.231607] Key type .llcrypt registered [ 5392.302990] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 5392.970144] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5392.978151] alg: No test for adler32 (adler32-zlib) [ 5393.971583] Lustre: Lustre: Build Version: 2.16.58_106_gac70ede [ 5394.126564] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 5395.759212] Key type lgssc registered [ 5396.623414] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5402.864507] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5402.883758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5404.321108] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5407.783449] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 5413.873860] LustreError: 196911:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5413.906566] LustreError: 196911:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5415.879128] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5416.255816] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5419.957714] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 5425.718691] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5430.439587] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5437.665300] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5438.036901] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5444.053251] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 5449.335397] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5449.827839] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5456.865400] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5456.868352] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5456.897781] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5457.853710] Lustre: lustre-OST0000: Not available for connect from 192.168.201.8@tcp (stopping) [ 5459.943703] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5459.944918] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5459.946796] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5459.971764] Lustre: Skipped 1 previous similar message [ 5462.794434] LustreError: 199158:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5462.976139] Lustre: server umount lustre-OST0000 complete [ 5468.219183] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff991685245000.ost_server_uuid 50 [ 5469.611060] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5482.980375] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5482.984203] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5482.995880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5485.029721] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5488.540282] LustreError: 199528:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5488.546904] LustreError: 199528:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5488.655851] Lustre: server umount lustre-MDT0000 complete [ 5490.144825] LustreError: 196912:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5490.161205] LustreError: 196912:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5492.132728] LustreError: 196892:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762495236 with bad export cookie 1203571094303114628 [ 5492.138355] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5492.146622] LustreError: 196892:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5492.260866] LustreError: 199730:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5492.263529] LustreError: 199730:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5492.529575] Lustre: server umount lustre-MDT0001 complete [ 5502.323043] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 01:00:44 (1762495244) [ 5509.503800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5510.092717] LustreError: 200402:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5510.171718] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5513.427527] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 5515.232962] LustreError: 200402:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5520.355370] LustreError: 200402:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5520.663363] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5520.968651] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5524.546909] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 5529.915492] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5533.852436] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5540.397433] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5540.809216] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5545.771531] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 5551.337110] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5555.223877] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5558.880400] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5559.010228] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5559.233154] Lustre: lustre-OST0001: new disk, initializing [ 5559.240868] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5559.336635] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5565.514193] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5565.527159] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5565.682871] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5565.709503] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 5573.207400] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5576.165777] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5576.172885] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5576.190892] Lustre: Skipped 1 previous similar message [ 5576.203017] Lustre: Skipped 3 previous similar messages [ 5581.885097] LustreError: 203619:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5581.897384] LustreError: 203619:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5581.979751] Lustre: server umount lustre-OST0000 complete [ 5584.297330] LustreError: 202011:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5588.056442] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5588.268865] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5592.256922] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5592.460818] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5601.769394] LustreError: 202012:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5601.795774] LustreError: 202012:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 5606.882925] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5606.898084] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5606.905743] Lustre: Skipped 1 previous similar message [ 5606.918332] Lustre: Skipped 4 previous similar messages [ 5611.452967] LustreError: 204344:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 5611.466335] LustreError: 204344:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5611.710043] Lustre: server umount lustre-OST0001 complete [ 5621.371848] LustreError: 204545:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 5621.374949] LustreError: 204545:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5621.480951] Lustre: server umount lustre-MDT0000 complete [ 5625.395478] LustreError: 202924:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762495369 with bad export cookie 1203571094303115699 [ 5625.399814] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5625.420845] LustreError: 202924:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5625.949030] Lustre: server umount lustre-MDT0001 complete [ 5632.065204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5635.717589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5639.280115] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5642.538518] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5649.192445] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 01:03:12 (1762495392) [ 5655.401098] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5655.685510] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5655.742777] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5655.894590] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5659.232963] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 5666.843351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5666.918195] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5666.957694] Lustre: 206243:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5671.410414] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 5678.919803] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5684.608547] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5691.229399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5691.358177] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5696.383128] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 5699.157698] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 5700.709720] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5705.732392] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5705.782130] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5707.477484] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 5710.831055] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 5715.759963] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5719.520684] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5719.523917] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5719.531397] Lustre: Skipped 2 previous similar messages [ 5719.536892] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5719.544990] Lustre: Skipped 3 previous similar messages [ 5724.359342] LustreError: 209443:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5724.373972] LustreError: 209443:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 5724.409169] Lustre: server umount lustre-OST0000 complete [ 5726.638780] LustreError: 208924:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5726.658318] LustreError: 208924:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 5743.584359] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5743.588244] LustreError: Skipped 2 previous similar messages [ 5745.588190] Lustre: server umount lustre-OST0001 complete [ 5753.829579] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5753.835050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5753.847832] Lustre: Skipped 4 previous similar messages [ 5753.863820] Lustre: Skipped 6 previous similar messages [ 5755.577696] Lustre: server umount lustre-MDT0000 complete [ 5759.240501] LustreError: 206237:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762495503 with bad export cookie 1203571094303117106 [ 5759.250836] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5759.255230] LustreError: 206237:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5759.452466] LustreError: 210070:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 5759.476057] LustreError: 210070:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 5759.704332] Lustre: server umount lustre-MDT0001 complete [ 5765.489846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5769.131850] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5772.639245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5775.985587] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5781.251868] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 01:05:24 (1762495524) [ 5795.571719] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_hostid [ 5805.635724] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 5815.116892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5821.774523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5827.692321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5835.452558] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5843.008511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5843.095688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5843.340226] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5843.377351] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5843.466980] Lustre: lustre-MDT0000: new disk, initializing [ 5843.581116] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5843.585764] Lustre: Skipped 3 previous similar messages [ 5843.598266] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5847.308600] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 5856.818954] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5856.876830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5856.948446] Lustre: 213895:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5857.019713] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5857.021870] Lustre: Skipped 1 previous similar message [ 5857.093798] Lustre: lustre-MDT0001: new disk, initializing [ 5857.228565] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5857.237713] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5860.731335] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 5868.698045] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5872.532442] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5878.727976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5878.786356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5879.058810] Lustre: lustre-OST0000: new disk, initializing [ 5879.068928] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5885.137414] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 5885.504196] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5885.516769] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5885.614356] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5891.781641] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5894.913807] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5895.119167] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5898.594652] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5898.783905] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5901.280953] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5901.285983] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5901.299801] LustreError: Skipped 1 previous similar message [ 5901.302202] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5901.306860] Lustre: Skipped 1 previous similar message [ 5901.315911] Lustre: Skipped 1 previous similar message [ 5906.545100] LustreError: 216831:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5906.550721] LustreError: 216831:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 5906.688195] Lustre: server umount lustre-OST0000 complete [ 5915.779983] Lustre: server umount lustre-MDT0000 complete [ 5916.640440] LustreError: 213907:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5916.655191] LustreError: 213907:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [ 5918.981105] LustreError: 213887:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762495663 with bad export cookie 1203571094303118884 [ 5918.981407] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5919.025912] LustreError: 213887:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5919.338628] Lustre: server umount lustre-MDT0001 complete [ 5926.610952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5931.446331] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 5938.880994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5941.916458] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 5946.969791] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5950.546809] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5956.139630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5961.392728] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 5966.101864] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5970.068781] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5972.411338] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 5975.595581] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5975.767380] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5979.106837] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5983.449606] Lustre: server umount lustre-OST0000 complete [ 5987.683991] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5990.809623] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6019.346749] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6019.692340] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6019.700916] Lustre: Skipped 5 previous similar messages [ 6021.737342] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6021.744149] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6024.555864] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 6029.277957] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6031.973903] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6031.995551] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6038.187745] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 7 sec [ 6042.591985] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6042.596656] LustreError: Skipped 1 previous similar message [ 6042.598726] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6042.603337] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6042.603344] Lustre: Skipped 7 previous similar messages [ 6042.630702] Lustre: Skipped 5 previous similar messages [ 6046.779148] LustreError: 222112:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6046.793904] LustreError: 222112:0:(obd_class.h:479:obd_check_dev()) Skipped 21 previous similar messages [ 6047.001272] Lustre: server umount lustre-OST0000 complete [ 6057.954024] LustreError: 217714:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6057.965883] LustreError: 217714:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6058.812383] Lustre: server umount lustre-MDT0000 complete [ 6062.366707] LustreError: 217696:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762495806 with bad export cookie 1203571094303119773 [ 6062.389065] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6062.717762] Lustre: server umount lustre-MDT0001 complete [ 6069.905568] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 01:10:12 (1762495812) [ 6075.239661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6078.504875] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 6084.890554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6088.109749] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 6092.336101] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6095.635161] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6102.171878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6108.445284] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 6114.380886] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6118.326476] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6121.739605] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 6126.231593] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6126.448655] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6134.440923] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6134.510918] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6134.604324] Lustre: lustre-OST0001: new disk, initializing [ 6134.608701] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6141.555981] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 6141.587416] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6141.603630] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6141.705892] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6148.916960] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6153.142907] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6153.348656] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6156.627210] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6156.806744] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6162.410884] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6162.416868] LustreError: Skipped 2 previous similar messages [ 6164.565474] Lustre: server umount lustre-OST0001 complete [ 6169.367151] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6169.553596] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6173.439491] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6173.640415] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6200.697302] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6205.829131] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 6205.935253] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6205.980083] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6205.984603] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6211.984748] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6213.869653] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6222.166793] Lustre: server umount lustre-OST0001 complete [ 6246.414653] Lustre: server umount lustre-OST0000 complete [ 6256.609835] Lustre: server umount lustre-MDT0000 complete [ 6259.690463] LustreError: 223167:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762496004 with bad export cookie 1203571094303120739 [ 6259.691274] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6259.708203] LustreError: 223167:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 6266.115885] Lustre: server umount lustre-MDT0001 complete [ 6271.050261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6273.700210] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6277.069243] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6280.102144] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6286.650613] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 01:13:49 (1762496029) [ 6292.764922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6292.945241] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6292.969897] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6293.110630] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6293.113645] Lustre: Skipped 5 previous similar messages [ 6296.197705] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 6303.610287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6303.817663] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6303.842594] Lustre: 231192:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6308.496626] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 6314.271885] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6318.776957] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6324.446656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6324.583462] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6329.597193] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 6335.381970] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6344.063615] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6344.221632] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6350.154435] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 6354.660128] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6356.357431] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6360.364905] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6360.614855] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6364.257308] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6364.443482] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6368.611990] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff991698d0d800.ost_server_uuid 50 [ 6370.052660] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff991698d0d800.ost_server_uuid in FULL state after 0 sec [ 6371.531966] Lustre: Permanently deactivating lustre-OST0001 [ 6371.549993] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6375.308343] Lustre: Permanently reactivating lustre-OST0001 [ 6383.583953] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6383.590457] LustreError: Skipped 2 previous similar messages [ 6383.592941] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6383.619068] Lustre: Skipped 10 previous similar messages [ 6383.624278] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6383.633269] Lustre: Skipped 15 previous similar messages [ 6394.341213] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6394.384955] LustreError: 235179:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 6394.388757] LustreError: 235179:0:(obd_class.h:479:obd_check_dev()) Skipped 47 previous similar messages [ 6394.422467] Lustre: server umount lustre-OST0001 complete [ 6399.463300] LustreError: 232818:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6399.495593] LustreError: 232818:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 34 previous similar messages [ 6404.385120] Lustre: server umount lustre-OST0000 complete [ 6413.430336] Lustre: server umount lustre-MDT0000 complete [ 6415.837674] LustreError: 231184:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762496160 with bad export cookie 1203571094303122027 [ 6415.844239] LustreError: 231184:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6415.846839] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6416.084021] Lustre: server umount lustre-MDT0001 complete [ 6420.064091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6422.883966] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6426.442174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6429.610253] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6435.130898] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 01:16:18 (1762496178) [ 6435.859362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6441.087828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6441.348658] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6441.374113] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6441.381081] Lustre: Skipped 5 previous similar messages [ 6444.840575] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 6451.690236] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6451.775974] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6451.806321] Lustre: 237342:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6454.975922] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 6459.450300] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6462.502458] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6467.430882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6467.492300] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6467.629128] Lustre: Permanently deactivating lustre-OST0000 [ 6467.647218] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6472.323468] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 6475.814396] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6477.121813] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6482.753385] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6485.250558] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6487.394695] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 6491.856326] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6493.186829] Lustre: Permanently reactivating lustre-OST0000 [ 6493.202243] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6493.205566] Lustre: Skipped 2 previous similar messages [ 6499.922874] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6499.929896] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6499.947831] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6499.952397] Lustre: Skipped 1 previous similar message [ 6500.000589] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6500.033355] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6506.389382] Lustre: server umount lustre-OST0001 complete [ 6516.485275] Lustre: server umount lustre-OST0000 complete [ 6526.664665] Lustre: server umount lustre-MDT0000 complete [ 6530.161301] LustreError: 240491:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762496274 with bad export cookie 1203571094303123672 [ 6530.164171] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6530.175107] LustreError: 240491:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6530.563343] Lustre: server umount lustre-MDT0001 complete [ 6541.101417] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 6543.119318] Key type lgssc unregistered [ 6543.419756] LNet: 241739:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6543.431672] LNetError: 241739:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6543.455109] LNet: Removed LNI 192.168.201.108@tcp [ 6544.147872] Key type .llcrypt unregistered [ 6544.150182] Key type ._llcrypt unregistered [ 6568.531254] Key type ._llcrypt registered [ 6568.535806] Key type .llcrypt registered [ 6568.682281] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 6569.788688] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6569.893518] alg: No test for adler32 (adler32-zlib) [ 6570.928865] Lustre: Lustre: Build Version: 2.16.58_106_gac70ede [ 6571.184752] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 6572.831176] Key type lgssc registered [ 6573.890064] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6577.295099] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6584.252953] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6584.285136] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6585.826970] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6589.695300] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 6596.068892] LustreError: 243192:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6596.080241] LustreError: 243192:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6597.197530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6597.278140] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6597.386293] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6597.418540] Lustre: 243181:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6597.427630] Lustre: Permanently deactivating lustre-MDT0001 [ 6597.441112] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6597.705551] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6600.934861] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 6601.233863] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6601.242659] LustreError: 243222:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 6601.259332] LustreError: 243222:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 6601.276506] LustreError: 243222:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 15, retries 1, failed: rc = -108 [ 6605.951952] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6609.580543] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6615.179132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6615.442840] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6620.238402] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 6622.636508] LustreError: 244811:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6625.100260] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6627.822537] LustreError: 244811:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6630.786110] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6630.907509] LustreError: 244811:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6630.915341] LustreError: 244811:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6630.917212] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 6631.102074] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6635.338918] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 6636.529178] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 6636.531057] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 6640.638606] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6645.419667] Lustre: Permanently reactivating lustre-MDT0001 [ 6645.458897] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 6645.472526] Lustre: Skipped 2 previous similar messages [ 6649.812540] Lustre: lustre-MDT0001: Client acaa7626-8dde-4823-8d35-396c0f08f6c2 (at 192.168.201.8@tcp) reconnecting [ 6654.883580] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6657.795188] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6664.127961] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6664.376212] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6668.683375] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6668.868727] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6671.379289] Lustre: Permanently deactivating lustre-MDT0001 [ 6671.399668] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6671.403203] Lustre: Skipped 2 previous similar messages [ 6680.703268] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6687.716958] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6687.719650] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6687.725540] Lustre: Skipped 2 previous similar messages [ 6687.736684] Lustre: Skipped 3 previous similar messages [ 6691.495833] LustreError: 247264:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 6691.760427] Lustre: server umount lustre-MDT0000 complete [ 6692.841449] LustreError: 243186:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6692.850456] LustreError: 243186:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6695.229632] LustreError: 243172:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762496439 with bad export cookie 4502979470133781616 [ 6695.234510] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6695.252315] LustreError: 243172:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6695.415461] LustreError: 247465:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 6695.428083] LustreError: 247465:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6695.691544] Lustre: server umount lustre-MDT0001 complete [ 6702.021099] LustreError: 247666:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 6702.036358] LustreError: 247666:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6702.209932] Lustre: server umount lustre-OST0000 complete [ 6713.091301] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 01:20:55 (1762496455) [ 6722.493263] LustreError: 248673:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6722.509358] LustreError: 248673:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 6722.661792] Lustre: server umount lustre-OST0001 complete [ 6728.309573] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_hostid [ 6738.151652] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 6748.022192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6754.339083] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6759.836269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6764.985356] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6772.286603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6772.405576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6772.677461] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6772.684357] Lustre: Skipped 2 previous similar messages [ 6772.713949] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6772.821266] Lustre: lustre-MDT0000: new disk, initializing [ 6772.950096] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6772.989956] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6777.656641] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 6787.907933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6788.002544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6788.079791] Lustre: 250859:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6788.133946] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6788.137144] Lustre: Skipped 1 previous similar message [ 6788.210110] Lustre: lustre-MDT0001: new disk, initializing [ 6788.318350] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6788.360723] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6788.377701] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6791.446578] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 6797.815222] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6801.143184] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6806.964049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6807.007515] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6807.239317] Lustre: lustre-OST0000: new disk, initializing [ 6807.247397] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6807.356743] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6812.399599] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 6816.319539] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6816.332861] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6816.400365] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6818.937853] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6821.252877] LustreError: 250865:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6828.902380] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6828.955197] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6829.092535] Lustre: lustre-OST0001: new disk, initializing [ 6829.095694] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6831.335504] LustreError: 250865:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6831.373508] LustreError: 250865:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6834.138911] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 6834.759648] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6834.779881] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6834.865588] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6841.457710] LustreError: 250865:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6841.831328] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6843.528147] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6845.408640] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6845.411850] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6845.415770] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6845.419157] Lustre: Skipped 2 previous similar messages [ 6845.426673] Lustre: Skipped 1 previous similar message [ 6849.991401] Lustre: lustre-OST0001: Not available for connect from 192.168.201.8@tcp (stopping) [ 6851.479855] LustreError: 254352:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 6851.499919] LustreError: 254352:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6851.664637] Lustre: server umount lustre-OST0001 complete [ 6855.084040] LustreError: 252590:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6871.009389] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6871.019827] Lustre: Skipped 1 previous similar message [ 6871.022028] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6871.032709] Lustre: Skipped 2 previous similar messages [ 6874.230356] LustreError: 254575:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6874.233617] LustreError: 254575:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6874.431058] Lustre: server umount lustre-OST0000 complete [ 6880.737714] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6880.741250] LustreError: Skipped 1 previous similar message [ 6880.743203] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6880.777317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6880.780501] Lustre: Skipped 1 previous similar message [ 6884.730558] Lustre: server umount lustre-MDT0000 complete [ 6886.371647] LustreError: 252108:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6886.386268] LustreError: 252108:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 6889.038788] LustreError: 250851:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762496633 with bad export cookie 4502979470133784213 [ 6889.041401] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6889.052331] LustreError: 250851:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6895.324523] LustreError: 254979:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 6895.346305] LustreError: 254979:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 6895.703956] Lustre: server umount lustre-MDT0001 complete [ 6906.391425] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 6909.095437] Key type lgssc unregistered [ 6909.394449] LNet: 255512:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6909.410137] LNetError: 255512:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6909.427538] LNet: Removed LNI 192.168.201.108@tcp [ 6910.159361] Key type .llcrypt unregistered [ 6910.161654] Key type ._llcrypt unregistered [ 6941.034226] Key type ._llcrypt registered [ 6941.035554] Key type .llcrypt registered [ 6941.190716] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_hostid [ 6956.181678] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 6958.434647] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6958.469202] alg: No test for adler32 (adler32-zlib) [ 6959.635853] Lustre: Lustre: Build Version: 2.16.58_106_gac70ede [ 6959.816049] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 6961.463167] Key type lgssc registered [ 6962.771964] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6973.837157] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6979.770489] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6985.290108] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6990.702692] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6997.344510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6997.400679] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6997.435081] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6998.697778] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6998.735194] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6998.797918] Lustre: lustre-MDT0000: new disk, initializing [ 6998.894875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6998.913367] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7002.267539] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 7011.738813] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7011.814681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7011.870442] Lustre: 259465:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7011.919061] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7011.921956] Lustre: Skipped 1 previous similar message [ 7011.979486] Lustre: lustre-MDT0001: new disk, initializing [ 7012.071168] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7012.150043] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7012.157784] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7016.165724] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 7023.017387] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7026.637621] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7033.138562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7033.206565] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7033.420150] Lustre: lustre-OST0000: new disk, initializing [ 7033.429860] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7033.500287] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7038.785457] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 7042.600678] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7042.607260] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7042.713170] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7045.128944] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7048.434308] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7048.648531] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7052.438985] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7052.670437] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7057.889270] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7057.901461] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7057.917195] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7058.409885] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7058.414465] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7058.419875] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7058.444988] Lustre: Skipped 1 previous similar message [ 7060.408472] LustreError: 262401:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7060.575150] Lustre: server umount lustre-OST0000 complete [ 7068.640766] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7068.641694] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7068.656369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7068.665098] Lustre: Skipped 1 previous similar message [ 7069.726692] LustreError: 262605:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7069.732205] LustreError: 262605:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7069.827421] Lustre: server umount lustre-MDT0000 complete [ 7072.603555] LustreError: 259457:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762496817 with bad export cookie 1746772382037557807 [ 7072.613333] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7072.615789] LustreError: 259457:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7072.723424] LustreError: 262806:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7072.726576] LustreError: 262806:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7072.938634] Lustre: server umount lustre-MDT0001 complete [ 7080.003530] Lustre: DEBUG MARKER: == conf-sanity test 52: check recovering objects from lost+found ========================================================== 01:27:03 (1762496823) [ 7084.643476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7085.015415] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7087.770339] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 7090.145637] LustreError: 263475:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7090.157974] LustreError: 263475:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7093.469849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7093.716572] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7096.955330] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 7102.031838] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7105.209897] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7110.902380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7111.297397] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7116.735131] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 7122.048236] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7186.405726] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7186.406112] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7186.411952] LustreError: Skipped 1 previous similar message [ 7186.413775] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7186.422293] Lustre: Skipped 2 previous similar messages [ 7186.433653] Lustre: Skipped 1 previous similar message [ 7189.557724] LustreError: 265823:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7189.563271] LustreError: 265823:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7189.713141] Lustre: server umount lustre-OST0000 complete [ 7193.754875] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7206.084225] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7206.424926] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7212.369599] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 7218.731748] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7225.832983] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7225.856069] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7225.856886] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7225.897989] Lustre: Skipped 1 previous similar message [ 7261.670532] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7261.671422] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7261.688302] LustreError: Skipped 1 previous similar message [ 7261.693164] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7264.310637] LustreError: 267492:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7264.321154] LustreError: 267492:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7264.588510] Lustre: server umount lustre-OST0000 complete [ 7271.905113] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7271.905113] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7271.905126] Lustre: Skipped 1 previous similar message [ 7271.906157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7271.922170] Lustre: Skipped 2 previous similar messages [ 7274.569548] LustreError: 267693:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7274.576531] LustreError: 267693:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7274.786077] Lustre: server umount lustre-MDT0000 complete [ 7277.026675] LustreError: 263476:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7277.045946] LustreError: 263476:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7278.853735] LustreError: 263456:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762497023 with bad export cookie 1746772382037558689 [ 7278.857602] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7278.875769] LustreError: 263456:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7279.425591] Lustre: server umount lustre-MDT0001 complete [ 7289.576397] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 7292.409537] Key type lgssc unregistered [ 7292.704136] LNet: 268426:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7292.708826] LNetError: 268426:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7292.722505] LNet: Removed LNI 192.168.201.108@tcp [ 7293.541597] Key type .llcrypt unregistered [ 7293.543328] Key type ._llcrypt unregistered [ 7313.721433] Key type ._llcrypt registered [ 7313.724523] Key type .llcrypt registered [ 7313.867679] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 7314.843620] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7314.858953] alg: No test for adler32 (adler32-zlib) [ 7316.003340] Lustre: Lustre: Build Version: 2.16.58_106_gac70ede [ 7316.230191] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 7317.919647] Key type lgssc registered [ 7319.158155] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7326.919917] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7326.957622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7328.489321] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7332.407323] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 7337.952189] LustreError: 269804:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7337.963462] LustreError: 269804:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7341.710812] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7342.193137] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7346.847301] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 7352.955639] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7356.323327] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7362.795238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7363.204839] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7368.648170] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 7373.421124] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7373.896959] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7398.368137] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7398.373917] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7398.400296] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7399.395525] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7399.397670] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7399.402196] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7399.418076] Lustre: Skipped 1 previous similar message [ 7402.036439] LustreError: 272826:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7402.336994] Lustre: server umount lustre-OST0000 complete [ 7408.610173] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7408.617568] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7408.625185] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7413.688995] LustreError: 273029:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7413.698231] LustreError: 273029:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7413.791891] Lustre: server umount lustre-MDT0000 complete [ 7414.763633] LustreError: 270920:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7414.792597] LustreError: 270920:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7417.533791] LustreError: 273030:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762497161 with bad export cookie 9188590932017190664 [ 7417.543567] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7417.546240] LustreError: 273030:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7417.708587] LustreError: 273232:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7417.712620] LustreError: 273232:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7417.940624] Lustre: server umount lustre-MDT0001 complete [ 7427.073886] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 7429.636503] Key type lgssc unregistered [ 7429.899426] LNet: 273764:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7429.909137] LNetError: 273764:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7429.928721] LNet: Removed LNI 192.168.201.108@tcp [ 7430.633103] Key type .llcrypt unregistered [ 7430.635969] Key type ._llcrypt unregistered [ 7446.292764] Key type ._llcrypt registered [ 7446.295351] Key type .llcrypt registered [ 7446.385562] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 7447.365319] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7447.458091] alg: No test for adler32 (adler32-zlib) [ 7448.644891] Lustre: Lustre: Build Version: 2.16.58_106_gac70ede [ 7448.875764] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 7450.519187] Key type lgssc registered [ 7451.703503] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7460.066771] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7460.080078] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7461.609615] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7466.727906] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 7471.073675] LustreError: 274967:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7471.114146] LustreError: 274967:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7476.198085] LustreError: 274967:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7476.305923] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7477.076400] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7481.590265] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 7487.609362] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7491.528963] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7499.177056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7499.438397] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7504.731646] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 7506.631435] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7510.569733] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7522.272171] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7522.276915] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7522.279636] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7522.297239] Lustre: Skipped 1 previous similar message [ 7523.713114] LustreError: 277359:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7523.950643] Lustre: server umount lustre-OST0000 complete [ 7528.418266] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7528.421326] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7528.441536] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7528.452055] Lustre: Skipped 1 previous similar message [ 7532.513759] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7532.516727] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7534.320654] LustreError: 277560:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7534.331068] LustreError: 277560:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7534.441154] Lustre: server umount lustre-MDT0000 complete [ 7537.638355] LustreError: 274962:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7537.657889] LustreError: 274962:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7538.299180] LustreError: 274948:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762497282 with bad export cookie 3280281958012143940 [ 7538.305843] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7538.494080] LustreError: 277763:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7538.503355] LustreError: 277763:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7538.748568] Lustre: server umount lustre-MDT0001 complete [ 7549.587796] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 7552.363332] Key type lgssc unregistered [ 7552.773715] LNet: 278294:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7552.780258] LNetError: 278294:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7552.805103] LNet: Removed LNI 192.168.201.108@tcp [ 7553.822649] Key type .llcrypt unregistered [ 7553.824527] Key type ._llcrypt unregistered [ 7578.281095] Key type ._llcrypt registered [ 7578.282984] Key type .llcrypt registered [ 7578.372223] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 7579.236122] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7579.407966] alg: No test for adler32 (adler32-zlib) [ 7580.538045] Lustre: Lustre: Build Version: 2.16.58_106_gac70ede [ 7580.777654] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 7582.455176] Key type lgssc registered [ 7583.568074] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7592.086564] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7592.131287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7593.794956] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7597.999196] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 7604.193450] LustreError: 279690:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7604.211917] LustreError: 279690:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7606.828277] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7607.410109] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7612.132539] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 7617.769045] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7621.885342] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7629.215350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7629.640375] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7635.822380] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 7638.848705] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7641.456934] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7664.614950] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7664.615377] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7664.621867] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7669.729503] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7669.735041] Lustre: Skipped 1 previous similar message [ 7670.340529] LustreError: 282708:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7670.548829] Lustre: server umount lustre-OST0000 complete [ 7674.852781] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7674.854937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7674.882260] Lustre: Skipped 1 previous similar message [ 7674.883408] Lustre: Skipped 2 previous similar messages [ 7679.971591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7679.988288] Lustre: Skipped 2 previous similar messages [ 7685.088829] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7685.091661] Lustre: Skipped 1 previous similar message [ 7688.159223] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7688.303496] LustreError: 282911:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7688.316297] LustreError: 282911:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7688.397020] Lustre: server umount lustre-MDT0000 complete [ 7690.228877] LustreError: 280403:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7690.264321] LustreError: 280403:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7692.634762] LustreError: 279670:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762497437 with bad export cookie 16016486131988364109 [ 7692.641828] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7692.653716] LustreError: 279670:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7692.803281] LustreError: 283113:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7692.812916] LustreError: 283113:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7693.109523] Lustre: server umount lustre-MDT0001 complete [ 7705.483837] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 7708.735792] Key type lgssc unregistered [ 7709.153707] LNet: 283649:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7709.169706] LNetError: 283649:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7709.199845] LNet: Removed LNI 192.168.201.108@tcp [ 7710.149642] Key type .llcrypt unregistered [ 7710.151501] Key type ._llcrypt unregistered [ 7726.382348] Key type ._llcrypt registered [ 7726.383766] Key type .llcrypt registered [ 7726.461351] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 7727.179787] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7727.189660] alg: No test for adler32 (adler32-zlib) [ 7728.165746] Lustre: Lustre: Build Version: 2.16.58_106_gac70ede [ 7728.352353] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 7730.007872] Key type lgssc registered [ 7731.000235] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7738.443781] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7738.481172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7740.010265] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7743.342707] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 7745.504802] LustreError: 284876:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7745.535089] LustreError: 284876:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7749.602023] LustreError: 284876:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7751.022888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7751.261871] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7755.084621] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 7760.653601] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7764.626310] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7770.195795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7770.506283] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7774.966668] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 7779.923663] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7784.907597] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 7795.170707] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7795.178186] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7795.188187] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7800.289982] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7800.307743] Lustre: Skipped 2 previous similar messages [ 7800.371281] LustreError: 287279:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7800.512853] Lustre: server umount lustre-OST0000 complete [ 7805.410242] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7805.419579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7805.432769] Lustre: Skipped 2 previous similar messages [ 7805.438066] Lustre: Skipped 1 previous similar message [ 7810.532690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7810.545655] Lustre: Skipped 1 previous similar message [ 7815.650099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7815.671636] Lustre: Skipped 1 previous similar message [ 7818.207215] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7818.297328] LustreError: 287482:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7818.302632] LustreError: 287482:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7820.388446] LustreError: 287482:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 7820.391249] LustreError: 287482:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7820.430292] Lustre: server umount lustre-MDT0000 complete [ 7820.775513] LustreError: 284857:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7820.791934] LustreError: 284857:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7824.336817] LustreError: 284829:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762497568 with bad export cookie 14130919209491219481 [ 7824.344203] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7824.350572] LustreError: 284829:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7824.540089] LustreError: 287688:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7824.551190] LustreError: 287688:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 7825.009925] Lustre: server umount lustre-MDT0001 complete [ 7836.322534] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 7838.752168] Key type lgssc unregistered [ 7838.955627] LNet: 288221:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7838.965839] LNetError: 288221:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7839.973685] LNet: Removed LNI 192.168.201.108@tcp [ 7840.747393] Key type .llcrypt unregistered [ 7840.749139] Key type ._llcrypt unregistered [ 7851.752604] Key type ._llcrypt registered [ 7851.758252] Key type .llcrypt registered [ 7851.959761] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 7868.583159] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_hostid [ 7883.866172] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 7884.761764] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7884.772616] alg: No test for adler32 (adler32-zlib) [ 7885.778722] Lustre: Lustre: Build Version: 2.16.58_106_gac70ede [ 7886.057874] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 7887.727177] Key type lgssc registered [ 7888.752601] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7898.350392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7904.671254] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7910.728537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7917.558246] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7924.186848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7924.241140] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7924.249630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7925.466134] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7925.492719] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7925.577117] Lustre: lustre-MDT0000: new disk, initializing [ 7925.653198] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7925.672208] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7929.159524] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 7938.104092] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7938.149138] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7938.216441] Lustre: 291730:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7938.269445] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7938.278465] Lustre: Skipped 1 previous similar message [ 7938.347967] Lustre: lustre-MDT0001: new disk, initializing [ 7938.434292] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7938.483962] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7938.489198] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7941.493515] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 7947.606168] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7950.500285] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7955.336581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7955.383836] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7955.569720] Lustre: lustre-OST0000: new disk, initializing [ 7955.577240] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7955.640851] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7959.956976] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 7964.727445] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7964.737253] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7964.854113] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7966.705431] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7970.112277] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7970.270801] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7974.321912] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7974.523256] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7980.519028] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7980.523755] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7980.537835] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7980.538832] LustreError: Skipped 1 previous similar message [ 7980.546430] Lustre: Skipped 1 previous similar message [ 7982.454043] LustreError: 294668:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7982.626761] Lustre: server umount lustre-OST0000 complete [ 7989.728404] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7989.748939] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7989.758764] Lustre: Skipped 1 previous similar message [ 7989.765308] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7990.753451] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7990.779194] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7990.786714] Lustre: Skipped 1 previous similar message [ 7992.477885] LustreError: 294869:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7992.480483] LustreError: 294869:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7992.745301] Lustre: server umount lustre-MDT0000 complete [ 7997.518890] LustreError: 291723:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762497741 with bad export cookie 15494237880778822323 [ 7997.526717] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7997.539396] LustreError: 291723:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7997.725934] LustreError: 295071:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7997.741791] LustreError: 295071:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7997.945619] Lustre: server umount lustre-MDT0001 complete [ 8007.752365] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 01:42:30 (1762497750) [ 8014.142038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8014.640212] LustreError: 295743:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8014.663891] LustreError: 295743:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8014.744088] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8017.612563] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 8019.939753] LustreError: 295743:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8024.815552] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8025.089384] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8028.150610] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 8032.581872] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8036.532040] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8042.828803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8043.086320] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8048.425293] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 8053.451699] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8076.262279] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8076.282545] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8076.307676] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8078.908070] LustreError: 298019:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8078.911256] LustreError: 298019:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 8079.149787] Lustre: server umount lustre-OST0000 complete [ 8086.498687] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8086.522629] LustreError: Skipped 1 previous similar message [ 8086.524940] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8086.572326] Lustre: Skipped 1 previous similar message [ 8086.595613] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8086.598825] Lustre: Skipped 2 previous similar messages [ 8089.473733] LustreError: 298221:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8089.477159] LustreError: 298221:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8089.576946] Lustre: server umount lustre-MDT0000 complete [ 8092.656162] LustreError: 295738:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8092.679675] LustreError: 295738:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8093.654319] LustreError: 295724:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762497838 with bad export cookie 15494237880778823205 [ 8093.667761] LustreError: 295724:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8093.673514] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8094.018671] Lustre: server umount lustre-MDT0001 complete [ 8104.977771] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 8107.511823] Key type lgssc unregistered [ 8107.771062] LNet: 298953:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8107.778386] LNetError: 298953:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8107.792590] LNet: Removed LNI 192.168.201.108@tcp [ 8108.434655] Key type .llcrypt unregistered [ 8108.439679] Key type ._llcrypt unregistered [ 8132.253882] Key type ._llcrypt registered [ 8132.255439] Key type .llcrypt registered [ 8132.361870] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_hostid [ 8147.421402] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 8148.165136] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8148.261252] alg: No test for adler32 (adler32-zlib) [ 8149.225361] Lustre: Lustre: Build Version: 2.16.58_106_gac70ede [ 8149.368913] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 8151.023261] Key type lgssc registered [ 8151.962694] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8160.735580] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8166.581578] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8171.774559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8177.492323] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8182.051293] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8187.411629] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8194.039621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8194.127970] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8194.144326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8195.367761] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8195.430347] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8195.485532] Lustre: lustre-MDT0000: new disk, initializing [ 8195.559912] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8195.569918] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8199.089278] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 8208.169886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8208.222870] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8208.298895] Lustre: 302759:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8208.349363] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8208.352414] Lustre: Skipped 1 previous similar message [ 8208.414252] Lustre: lustre-MDT0001: new disk, initializing [ 8208.531155] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8208.577576] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8208.591284] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8211.958418] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 8219.460191] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8223.785578] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8230.524644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8230.574579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8230.763988] Lustre: lustre-OST2710: new disk, initializing [ 8230.768697] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8230.837266] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8234.927219] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 8236.570119] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8236.575815] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8236.657804] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8241.313764] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 8247.013263] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8247.092840] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8247.249567] Lustre: lustre-OST03e8: new disk, initializing [ 8247.252906] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8247.350521] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8252.571923] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 8257.092359] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8257.096743] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8257.180533] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8257.685589] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 8262.902776] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8263.116944] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8266.327358] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8266.465908] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8269.872620] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8270.122945] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8273.370866] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8273.591830] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8277.988589] Lustre: lustre-MDT0000-lwp-OST03e8: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8278.012815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8278.022766] Lustre: Skipped 2 previous similar messages [ 8278.040864] Lustre: Skipped 3 previous similar messages [ 8279.311500] LustreError: 307105:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8279.417504] Lustre: server umount lustre-MDT0000 complete [ 8282.304369] LustreError: 305460:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762498026 with bad export cookie 3748372977187598666 [ 8282.308827] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8282.318751] LustreError: 305460:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8282.423517] LustreError: 307306:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8282.433428] LustreError: 307306:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 8282.578283] Lustre: server umount lustre-MDT0001 complete [ 8296.256087] LustreError: 307506:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8296.258976] LustreError: 307506:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 8296.282563] Lustre: server umount lustre-OST2710 complete [ 8309.120606] LustreError: 307708:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 8309.128395] LustreError: 307708:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8309.290405] Lustre: server umount lustre-OST03e8 complete [ 8324.937235] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_hostid [ 8333.228665] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 8342.085209] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8348.073439] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8353.035431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8358.067527] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8363.544542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8363.593962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8363.742784] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8363.764817] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8363.831718] Lustre: lustre-MDT0000: new disk, initializing [ 8363.929028] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8363.945485] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8367.074956] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 8375.691416] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8375.740720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8375.819389] Lustre: 310506:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8375.866983] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8375.869574] Lustre: Skipped 1 previous similar message [ 8375.938835] Lustre: lustre-MDT0001: new disk, initializing [ 8376.027638] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8376.062023] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8376.066056] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8379.178735] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 8386.218777] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8389.988420] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8397.878336] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8397.946828] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8398.219085] Lustre: lustre-OST0000: new disk, initializing [ 8398.221892] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8398.349419] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8406.080272] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 8406.088743] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8406.092170] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8406.273788] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8414.633062] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8418.915481] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8419.174270] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8423.614559] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8423.801612] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8426.976944] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8426.983362] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8426.987781] Lustre: Skipped 2 previous similar messages [ 8431.680129] LustreError: 313450:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8431.713052] LustreError: 313450:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8431.939341] Lustre: server umount lustre-OST0000 complete [ 8437.223094] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8437.225205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8437.267719] Lustre: Skipped 1 previous similar message [ 8437.273689] Lustre: Skipped 1 previous similar message [ 8442.336269] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8442.345624] Lustre: Skipped 2 previous similar messages [ 8447.456657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8447.469760] Lustre: Skipped 1 previous similar message [ 8448.991140] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8449.089403] LustreError: 313651:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8449.096171] LustreError: 313651:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8449.236475] Lustre: server umount lustre-MDT0000 complete [ 8453.240382] LustreError: 312255:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762498197 with bad export cookie 3748372977187600346 [ 8453.246841] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8453.259427] LustreError: 312255:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8453.676414] Lustre: server umount lustre-MDT0001 complete [ 8462.987613] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 01:50:05 (1762498205) [ 8464.574943] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8466.403186] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 01:50:08 (1762498208) [ 8469.458694] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 8477.909881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8481.969780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8485.712945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8489.456124] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8490.620791] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8497.337799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8497.611220] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8497.651438] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8497.780754] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8501.041303] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 8507.997266] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8508.050902] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8508.071634] Lustre: 315985:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8512.038539] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 8517.632071] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8521.413534] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8528.967054] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8529.060176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8529.264900] Lustre: Denying initial registration attempt from nid 192.168.201.108@tcp, specified as failover [ 8529.282321] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 8529.289980] LustreError: 317601:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -99 [ 8529.293717] LustreError: 317601:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 8529.296777] LustreError: 317601:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 8529.507264] Lustre: server umount lustre-OST0000 complete [ 8529.509315] LustreError: 317601:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 8538.599309] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8538.628172] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8538.631327] Lustre: Skipped 1 previous similar message [ 8550.367115] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8550.492319] LustreError: 317969:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 8550.494722] LustreError: 317969:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 8550.604765] Lustre: server umount lustre-MDT0000 complete [ 8553.657996] LustreError: 315978:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762498298 with bad export cookie 3748372977187601228 [ 8553.669086] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8553.986413] Lustre: server umount lustre-MDT0001 complete [ 8562.997546] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 8565.293728] Key type lgssc unregistered [ 8565.544068] LNet: 318703:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8565.572696] LNetError: 318703:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8566.629626] LNet: Removed LNI 192.168.201.108@tcp [ 8567.404285] Key type .llcrypt unregistered [ 8567.405854] Key type ._llcrypt unregistered [ 8577.337417] Key type ._llcrypt registered [ 8577.339932] Key type .llcrypt registered [ 8577.431894] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 8578.223422] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8578.332146] alg: No test for adler32 (adler32-zlib) [ 8579.421883] Lustre: Lustre: Build Version: 2.16.58_106_gac70ede [ 8579.656756] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 8581.290104] Key type lgssc registered [ 8582.208799] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8588.110947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8591.740214] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8595.438382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8599.247830] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8600.352262] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8615.359041] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 8624.229100] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8624.250452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8625.553161] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8625.592172] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8625.808386] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8628.999594] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 8637.396568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8637.466553] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8637.497915] Lustre: 321475:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8637.711244] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8641.428158] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 8647.140404] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8651.200733] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8659.227687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8659.342860] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8659.588427] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8659.795825] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8666.213488] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 8672.205769] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8674.791580] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8674.792118] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8674.802752] LustreError: Skipped 1 previous similar message [ 8674.807826] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8674.819158] Lustre: Skipped 1 previous similar message [ 8676.360220] LustreError: 323760:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8676.684609] Lustre: server umount lustre-OST0000 complete [ 8679.906582] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8679.911297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8679.912579] Lustre: Skipped 1 previous similar message [ 8679.923769] Lustre: Skipped 2 previous similar messages [ 8685.025873] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8685.028880] Lustre: Skipped 1 previous similar message [ 8690.146022] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8690.150127] Lustre: Skipped 1 previous similar message [ 8694.239618] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8694.327089] LustreError: 323962:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8694.334393] LustreError: 323962:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8694.432290] Lustre: server umount lustre-MDT0000 complete [ 8697.983335] LustreError: 323761:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762498442 with bad export cookie 14336401246498278354 [ 8697.994615] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8698.019277] LustreError: 323761:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8698.187672] LustreError: 324165:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8698.191312] LustreError: 324165:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8698.495031] Lustre: server umount lustre-MDT0001 complete [ 8708.367458] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 8711.087653] Key type lgssc unregistered [ 8711.351496] LNet: 324698:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8711.372565] LNetError: 324698:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8711.393311] LNet: Removed LNI 192.168.201.108@tcp [ 8712.174246] Key type .llcrypt unregistered [ 8712.176306] Key type ._llcrypt unregistered [ 8735.273133] Key type ._llcrypt registered [ 8735.276379] Key type .llcrypt registered [ 8735.377850] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 8736.077023] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8736.130082] alg: No test for adler32 (adler32-zlib) [ 8737.175168] Lustre: Lustre: Build Version: 2.16.58_106_gac70ede [ 8737.407284] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 8739.088467] Key type lgssc registered [ 8740.049828] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8746.796655] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8746.818352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8748.166373] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8751.731935] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 8753.633038] LustreError: 326106:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8753.641949] LustreError: 326106:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8757.735425] LustreError: 326106:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8760.171180] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8760.572223] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8764.639900] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 8768.861651] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8771.602098] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8776.824531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8777.107949] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8781.565912] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 8786.964286] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8801.764207] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8801.776352] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8801.802349] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8803.821992] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8803.823330] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8803.832702] Lustre: Skipped 1 previous similar message [ 8803.841761] Lustre: Skipped 2 previous similar messages [ 8807.829224] LustreError: 328376:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8807.948737] Lustre: server umount lustre-MDT0000 complete [ 8808.928381] LustreError: 326106:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8808.942303] LustreError: 326106:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8811.319801] LustreError: 326087:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762498555 with bad export cookie 1106749574714977664 [ 8811.323073] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8811.337806] LustreError: 326087:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8814.054677] LustreError: 327289:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8814.055590] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8814.078721] LustreError: 327289:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8814.091225] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8814.094022] Lustre: Skipped 1 previous similar message [ 8816.112280] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8816.130549] Lustre: Skipped 1 previous similar message [ 8817.579001] LustreError: 328577:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 8817.629098] LustreError: 328577:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8818.023148] Lustre: server umount lustre-MDT0001 complete [ 8823.948232] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8831.499921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8836.578539] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xf5bf6dfd8338ae5 [ 8836.606217] Lustre: MGC192.168.201.108@tcp: Connection restored to 0@lo (at 0@lo) [ 8836.891143] LustreError: 327715:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8836.902863] LustreError: 327715:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8836.985265] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8837.107748] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 8840.960714] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 8846.186695] LustreError: 327716:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8846.190148] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8846.193874] LustreError: 327716:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8848.961349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8849.315088] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8853.211559] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 8854.507714] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8858.194448] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8862.060884] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8865.432093] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8867.028258] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8872.937634] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8872.942947] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8872.951574] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8874.981325] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8885.215413] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8885.220905] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8885.227388] Lustre: Skipped 5 previous similar messages [ 8885.268169] LustreError: 330885:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8885.278663] LustreError: 330885:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8885.474498] Lustre: server umount lustre-OST0000 complete [ 8890.337500] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8890.339917] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8890.346816] Lustre: Skipped 1 previous similar message [ 8894.620081] LustreError: 331087:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8894.625260] LustreError: 331087:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8894.714341] Lustre: server umount lustre-MDT0000 complete [ 8895.458980] LustreError: 329387:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8895.478527] LustreError: 329387:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8897.607379] LustreError: 326085:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762498642 with bad export cookie 1106749574714985189 [ 8897.616692] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8897.618431] LustreError: 326085:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8897.855933] Lustre: server umount lustre-MDT0001 complete [ 8905.445456] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 8907.578552] Key type lgssc unregistered [ 8907.818678] LNet: 331820:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8907.823579] LNetError: 331820:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8907.836727] LNet: Removed LNI 192.168.201.108@tcp [ 8908.441317] Key type .llcrypt unregistered [ 8908.444554] Key type ._llcrypt unregistered [ 8929.558619] Key type ._llcrypt registered [ 8929.559904] Key type .llcrypt registered [ 8929.676622] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 8930.938057] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8930.957194] alg: No test for adler32 (adler32-zlib) [ 8932.169824] Lustre: Lustre: Build Version: 2.16.58_106_gac70ede [ 8932.483284] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 8934.151272] Key type lgssc registered [ 8935.344785] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8944.497898] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8944.557362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8946.118540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8950.961149] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 8956.385082] LustreError: 333220:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8956.401178] LustreError: 333220:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8960.691920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8960.996304] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8967.046350] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 8975.144928] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8979.593227] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8982.936588] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8982.954073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8986.019072] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8986.026927] Lustre: Skipped 2 previous similar messages [ 8991.202315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8991.205019] Lustre: Skipped 1 previous similar message [ 8996.320939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8996.334450] Lustre: Skipped 1 previous similar message [ 8997.343242] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8997.391530] LustreError: 334614:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8997.446490] Lustre: server umount lustre-MDT0000 complete [ 9000.770755] LustreError: 333202:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762498745 with bad export cookie 1507108043097907437 [ 9000.778405] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9000.787408] LustreError: 333202:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9000.898166] LustreError: 334817:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 9000.902683] LustreError: 334817:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9001.093291] Lustre: server umount lustre-MDT0001 complete [ 9014.247659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9014.471191] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9014.509136] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9014.706363] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9018.739473] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 9028.731849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9028.941983] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9028.988467] Lustre: 335587:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9029.280886] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9033.564512] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 9039.351490] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9042.820518] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9048.918666] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9049.139858] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9049.148333] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9049.153715] LustreError: 337247:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 9049.157695] LustreError: 337247:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 9049.160269] LustreError: 337247:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 9049.256141] Lustre: server umount lustre-OST0000 complete [ 9049.265198] LustreError: 337247:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9056.290058] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9056.447560] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9056.616120] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9060.746740] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 9064.020769] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9065.092307] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9071.319406] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9071.405810] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9071.447511] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9071.511979] Lustre: lustre-OST0001: new disk, initializing [ 9071.516570] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9071.605146] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9076.452702] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 9080.661210] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 9081.392056] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9081.406794] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9081.524727] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9086.945394] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9086.948409] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9086.959333] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9086.962284] Lustre: Skipped 2 previous similar messages [ 9086.967854] Lustre: Skipped 1 previous similar message [ 9089.337836] LustreError: 339372:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 9089.345882] LustreError: 339372:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 9089.414900] Lustre: server umount lustre-OST0001 complete [ 9092.071601] LustreError: 337716:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9092.080054] LustreError: 337716:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9094.630717] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9094.632596] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9094.643224] LustreError: Skipped 2 previous similar messages [ 9097.192658] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9097.194834] LustreError: 337715:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9097.201496] Lustre: Skipped 3 previous similar messages [ 9097.210586] LustreError: 337715:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9098.489409] LustreError: 339575:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 9098.498281] LustreError: 339575:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9098.662402] Lustre: server umount lustre-OST0000 complete [ 9102.310387] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9102.316134] Lustre: Skipped 1 previous similar message [ 9116.127135] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9116.250862] LustreError: 339776:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 9116.259937] LustreError: 339776:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9116.351250] Lustre: server umount lustre-MDT0000 complete [ 9117.666471] LustreError: 335593:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9117.680122] LustreError: 335593:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9119.915871] LustreError: 335578:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762498864 with bad export cookie 1507108043097907997 [ 9119.919274] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9119.938090] LustreError: 335578:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 9120.358803] Lustre: server umount lustre-MDT0001 complete [ 9131.080936] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 9134.850735] Key type lgssc unregistered [ 9135.274710] LNet: 340522:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9135.290387] LNetError: 340522:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9135.302403] LNet: Removed LNI 192.168.201.108@tcp [ 9136.299561] Key type .llcrypt unregistered [ 9136.301308] Key type ._llcrypt unregistered [ 9165.277385] Key type ._llcrypt registered [ 9165.278630] Key type .llcrypt registered [ 9165.372775] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_hostid [ 9178.395726] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 9179.963766] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9179.994186] alg: No test for adler32 (adler32-zlib) [ 9181.243318] Lustre: Lustre: Build Version: 2.16.58_106_gac70ede [ 9181.577613] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 9183.319183] Key type lgssc registered [ 9184.632604] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9194.101994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9201.340140] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9206.411468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9212.406048] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9217.352626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9217.386188] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9217.396577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9218.541778] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9218.567976] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9218.654933] Lustre: lustre-MDT0000: new disk, initializing [ 9218.766369] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9218.801510] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9222.157716] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 9231.591755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9231.690750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9231.792367] Lustre: 344470:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9231.883719] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9231.887897] Lustre: Skipped 1 previous similar message [ 9231.986894] Lustre: lustre-MDT0001: new disk, initializing [ 9232.087889] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9232.125868] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9232.141052] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9235.262884] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 9242.359410] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9245.964286] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9251.955996] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9252.001480] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9252.206429] Lustre: lustre-OST0000: new disk, initializing [ 9252.216858] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9252.342947] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9257.797237] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 9258.060783] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9258.071401] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9258.172538] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9264.153497] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9267.519573] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9267.680951] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9270.569183] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9270.706671] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9273.828432] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9273.828882] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9273.833885] LustreError: Skipped 1 previous similar message [ 9273.842400] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9278.946740] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9278.951815] Lustre: Skipped 2 previous similar messages [ 9284.065734] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9284.068851] Lustre: Skipped 1 previous similar message [ 9286.111184] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9286.165140] LustreError: 347411:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 9286.275670] Lustre: server umount lustre-OST0000 complete [ 9293.794613] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9293.800336] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9293.816503] Lustre: Skipped 1 previous similar message [ 9293.827129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9295.469241] LustreError: 347614:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 9295.471840] LustreError: 347614:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9295.609672] Lustre: server umount lustre-MDT0000 complete [ 9298.471731] LustreError: 346217:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762499042 with bad export cookie 15810853891149989618 [ 9298.478226] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9298.483236] LustreError: 346217:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9298.576754] LustreError: 347817:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9298.580549] LustreError: 347817:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9298.742389] Lustre: server umount lustre-MDT0001 complete [ 9306.656237] Lustre: DEBUG MARKER: == conf-sanity test 60a: check mkfs.lustre --mkfsoptions -E -O options setting ========================================================== 02:04:09 (1762499049) [ 9311.342162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9316.428953] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9335.881907] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_hostid [ 9343.305778] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 9351.293701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9356.840376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9361.946470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9366.879951] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9371.945623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9372.005216] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9372.197129] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9372.229901] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9372.281792] Lustre: lustre-MDT0000: new disk, initializing [ 9372.394166] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9372.416510] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9375.423788] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 9383.818468] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9383.879067] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9383.962441] Lustre: 351595:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9384.022776] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9384.034582] Lustre: Skipped 1 previous similar message [ 9384.109399] Lustre: lustre-MDT0001: new disk, initializing [ 9384.192451] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9384.233345] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9384.245517] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9387.091145] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 9392.973290] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9396.735528] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9403.936375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9404.005811] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9404.260321] Lustre: lustre-OST0000: new disk, initializing [ 9404.263870] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9404.392218] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9409.011634] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 9413.721172] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9413.732361] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9413.854622] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9416.418231] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9420.003789] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9420.211861] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9424.192369] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9424.398114] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9429.472365] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9429.483754] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9429.486673] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9429.493897] Lustre: Skipped 2 previous similar messages [ 9429.505510] Lustre: Skipped 3 previous similar messages [ 9439.714508] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9439.716931] Lustre: Skipped 3 previous similar messages [ 9440.224434] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9440.261154] LustreError: 354529:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 9440.264526] LustreError: 354529:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 9440.442754] Lustre: server umount lustre-OST0000 complete [ 9444.834058] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9444.840632] Lustre: Skipped 1 previous similar message [ 9458.143083] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9458.241960] LustreError: 354734:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 9458.245135] LustreError: 354734:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9458.290054] Lustre: server umount lustre-MDT0000 complete [ 9461.608543] LustreError: 351587:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762499206 with bad export cookie 15810853891149990500 [ 9461.610189] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9461.620966] LustreError: 351587:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9461.917242] Lustre: server umount lustre-MDT0001 complete [ 9469.817507] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 02:06:52 (1762499212) [ 9477.164174] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 02:06:59 (1762499219) [ 9483.828821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9484.111861] LustreError: 355952:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9484.133276] LustreError: 355952:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9484.163840] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9486.934365] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 9489.378882] LustreError: 355952:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9493.344644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9497.048404] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 9501.425326] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9504.235417] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9509.265645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9514.150436] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 9518.761967] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9525.915436] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 9527.769267] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9529.338637] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9531.342708] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 9533.096904] Lustre: Failing over lustre-MDT0000 [ 9533.251117] LustreError: 358390:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9533.253977] LustreError: 358390:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 9533.346211] Lustre: server umount lustre-MDT0000 complete [ 9533.408600] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9533.409891] LustreError: 356737:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9533.427306] Lustre: Skipped 1 previous similar message [ 9533.454061] LustreError: 356737:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9535.954596] LustreError: 355946:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9541.031937] LustreError: 355946:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9541.047644] LustreError: 355946:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9551.193404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9551.303875] LustreError: 356737:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9551.317676] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9551.328680] LustreError: 356737:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 9551.602270] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9551.610686] Lustre: Skipped 2 previous similar messages [ 9551.670951] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9554.972571] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 9556.395633] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9556.987807] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9557.000955] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 9557.042255] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9562.588980] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9564.873804] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9569.025787] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 9570.884683] Lustre: Failing over lustre-MDT0000 [ 9571.818883] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.8@tcp (stopping) [ 9571.832399] Lustre: Skipped 6 previous similar messages [ 9572.322670] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9572.326944] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9572.343230] LustreError: Skipped 1 previous similar message [ 9572.343886] Lustre: Skipped 2 previous similar messages [ 9573.066963] LustreError: 359469:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9573.073549] LustreError: 359469:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9573.206235] Lustre: server umount lustre-MDT0000 complete [ 9576.874192] LustreError: 355946:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9591.972156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9592.129647] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9592.396426] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9595.576319] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 9597.364056] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9597.414781] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9597.422620] Lustre: Skipped 2 previous similar messages [ 9597.432277] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 9597.471345] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 9602.735692] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9604.318421] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9612.768096] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9612.772415] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9612.780925] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9612.780934] Lustre: Skipped 4 previous similar messages [ 9612.799393] Lustre: Skipped 2 previous similar messages [ 9615.982506] LustreError: 360506:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9615.984601] LustreError: 360506:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9616.091952] Lustre: server umount lustre-MDT0000 complete [ 9617.892691] LustreError: 358012:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9617.910951] LustreError: 358012:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 13 previous similar messages [ 9619.188315] LustreError: 355931:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762499363 with bad export cookie 15810853891149993195 [ 9619.188787] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9619.192400] LustreError: 355931:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9619.485463] Lustre: server umount lustre-MDT0001 complete [ 9633.159297] Lustre: server umount lustre-OST0000 complete [ 9649.935496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9650.468560] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9650.471908] Lustre: Skipped 1 previous similar message [ 9654.121569] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 9663.059778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9667.902551] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 9673.806683] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9678.334042] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9685.770061] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9692.800507] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 9695.290930] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 9698.879592] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9703.394251] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9703.399965] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9703.405765] Lustre: Skipped 1 previous similar message [ 9703.427993] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9703.431246] Lustre: Skipped 2 previous similar messages [ 9709.449099] LustreError: 363984:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9709.462889] LustreError: 363984:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 9709.782070] Lustre: server umount lustre-OST0000 complete [ 9720.029457] Lustre: server umount lustre-MDT0000 complete [ 9720.802919] LustreError: 362718:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9720.814928] LustreError: 362718:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 9723.674650] LustreError: 361715:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762499468 with bad export cookie 15810853891149993923 [ 9723.680133] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9723.686878] LustreError: 361715:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9723.952087] Lustre: server umount lustre-MDT0001 complete [ 9735.017365] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 9738.431852] Key type lgssc unregistered [ 9738.748159] LNet: 364919:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9738.754210] LNetError: 364919:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9738.779797] LNet: Removed LNI 192.168.201.108@tcp [ 9739.783105] Key type .llcrypt unregistered [ 9739.785413] Key type ._llcrypt unregistered [ 9766.937441] Key type ._llcrypt registered [ 9766.939048] Key type .llcrypt registered [ 9767.085666] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 9768.329672] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9768.454696] alg: No test for adler32 (adler32-zlib) [ 9769.648925] Lustre: Lustre: Build Version: 2.16.58_106_gac70ede [ 9769.855366] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 9771.543159] Key type lgssc registered [ 9772.888114] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9781.340807] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9781.371336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9782.878172] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9786.585770] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 9792.995695] LustreError: 366418:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9793.015464] LustreError: 366418:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9795.134268] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9795.484929] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9799.197426] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 9804.553735] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9807.478791] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9812.695414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9813.020196] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9817.616251] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 9822.304212] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9828.400293] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 9841.634147] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9841.642895] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9841.655646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9843.682977] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9843.685958] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9843.702198] Lustre: Skipped 1 previous similar message [ 9843.707915] Lustre: Skipped 1 previous similar message [ 9846.950567] LustreError: 368692:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9847.046369] Lustre: server umount lustre-MDT0000 complete [ 9848.805298] LustreError: 366413:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9848.826805] LustreError: 366413:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9850.941705] LustreError: 366399:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762499595 with bad export cookie 17814660597520610757 [ 9850.951054] LustreError: 366399:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9850.962340] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9851.158952] LustreError: 368893:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 9851.161143] LustreError: 368893:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9851.511629] Lustre: server umount lustre-MDT0001 complete [ 9865.341461] LustreError: 369095:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 9865.344928] LustreError: 369095:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9865.579325] Lustre: server umount lustre-OST0000 complete [ 9880.743794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9881.119041] LustreError: 369828:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9881.191956] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9886.326718] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 9886.709087] LustreError: 369828:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9890.784086] LustreError: 369828:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9895.843549] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9896.214932] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9899.871992] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 9904.630808] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9907.703383] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9913.176283] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9913.516498] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9918.665418] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 9923.296895] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9925.810427] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 9941.477412] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9941.480193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9941.491453] Lustre: Skipped 1 previous similar message [ 9941.495441] Lustre: Skipped 1 previous similar message [ 9945.378249] LustreError: 372177:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9945.396856] LustreError: 372177:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9945.470115] Lustre: server umount lustre-MDT0000 complete [ 9946.595443] LustreError: 371792:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9946.617529] LustreError: 371792:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9948.594311] LustreError: 372090:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762499693 with bad export cookie 17814660597520611863 [ 9948.601559] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9948.867316] Lustre: server umount lustre-MDT0001 complete [ 9962.231914] LustreError: 372581:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 9962.234591] LustreError: 372581:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 9962.398664] Lustre: server umount lustre-OST0000 complete [ 9975.152614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9975.534104] LustreError: 373411:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9975.603555] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9978.754165] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 9985.832758] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9989.434559] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 9993.860323] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9996.881843] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10003.214275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10008.468545] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [10013.239744] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10019.961146] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10030.561091] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10030.564523] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10030.584129] Lustre: Skipped 2 previous similar messages [10034.421282] LustreError: 375679:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10034.438119] LustreError: 375679:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10034.567967] Lustre: server umount lustre-OST0000 complete [10040.802893] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10040.826362] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10040.829269] Lustre: Skipped 2 previous similar messages [10043.933562] Lustre: server umount lustre-MDT0000 complete [10045.920113] LustreError: 373411:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10045.938238] LustreError: 373411:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10047.241091] LustreError: 375881:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762499791 with bad export cookie 17814660597520612892 [10047.245570] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10047.273887] LustreError: 375881:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 7 previous similar messages [10047.583996] Lustre: server umount lustre-MDT0001 complete [10056.976651] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [10059.581533] Key type lgssc unregistered [10059.867041] LNet: 376615:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10059.874763] LNetError: 376615:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10059.893616] LNet: Removed LNI 192.168.201.108@tcp [10060.579472] Key type .llcrypt unregistered [10060.585270] Key type ._llcrypt unregistered [10083.323081] Key type ._llcrypt registered [10083.324614] Key type .llcrypt registered [10083.441138] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [10084.286780] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10084.321728] alg: No test for adler32 (adler32-zlib) [10085.428923] Lustre: Lustre: Build Version: 2.16.58_106_gac70ede [10085.676505] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [10087.351191] Key type lgssc registered [10088.428758] Lustre: Echo OBD driver; http://www.lustre.org/ [10095.097188] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10095.124839] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10095.131858] LustreError: 378033:0:(osd_handler.c:8486:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10095.151997] LustreError: 378033:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000-osd failed (-22) [10095.156249] LustreError: 378033:0:(obd_mount.c:193:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10095.160512] LustreError: 378033:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10095.164628] LustreError: 378033:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [10101.445515] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: user_xattr,acl,no_mbcache,nodelalloc [10101.449696] LustreError: 378431:0:(osd_handler.c:8486:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10101.464128] LustreError: 378431:0:(obd_config.c:783:class_setup()) setup lustre-OST0000-osd failed (-22) [10101.468876] LustreError: 378431:0:(obd_mount.c:193:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10101.473609] LustreError: 378431:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10101.479299] LustreError: 378431:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [10117.113650] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [10119.584792] Key type lgssc unregistered [10119.889085] LNet: 379418:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10119.896348] LNetError: 379418:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10119.908713] LNet: Removed LNI 192.168.201.108@tcp [10120.667768] Key type .llcrypt unregistered [10120.671988] Key type ._llcrypt unregistered [10137.983821] Key type ._llcrypt registered [10137.988571] Key type .llcrypt registered [10138.116016] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_hostid [10151.993100] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [10152.565184] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10152.576483] alg: No test for adler32 (adler32-zlib) [10153.628327] Lustre: Lustre: Build Version: 2.16.58_106_gac70ede [10153.852138] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [10155.503190] Key type lgssc registered [10156.442282] Lustre: Echo OBD driver; http://www.lustre.org/ [10166.061848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10172.836903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10178.863089] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10184.238788] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10190.669912] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10190.708554] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10190.733316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10191.920361] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10191.945797] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10191.989511] Lustre: lustre-MDT0000: new disk, initializing [10192.088453] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10192.100212] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10196.640763] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [10206.340527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10206.392691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10206.491452] Lustre: 382574:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10206.552456] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10206.555139] Lustre: Skipped 1 previous similar message [10206.606884] Lustre: lustre-MDT0001: new disk, initializing [10206.716633] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10206.765602] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10206.782035] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10210.672535] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [10217.754520] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10221.403591] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10227.905273] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10227.971784] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10228.212609] Lustre: lustre-OST0000: new disk, initializing [10228.222090] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10228.302510] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10233.008033] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [10234.451746] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10234.456192] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10234.604938] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10239.579362] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10243.559686] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10243.789447] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10247.466938] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10247.659974] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10250.211355] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10250.213225] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10250.231361] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10255.335371] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10255.338857] Lustre: Skipped 1 previous similar message [10255.397306] LustreError: 385514:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [10255.580202] Lustre: server umount lustre-OST0000 complete [10260.450497] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10260.455508] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10260.469323] Lustre: Skipped 2 previous similar messages [10260.473735] Lustre: Skipped 2 previous similar messages [10265.292789] LustreError: 385716:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [10265.313812] LustreError: 385716:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10265.390208] Lustre: server umount lustre-MDT0000 complete [10268.811658] LustreError: 382565:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762500013 with bad export cookie 10968251548520957947 [10268.817524] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10268.835474] LustreError: 382565:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [10269.035299] LustreError: 385918:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10269.039323] LustreError: 385918:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10269.238764] Lustre: server umount lustre-MDT0001 complete [10276.980645] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [10278.355221] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 02:20:21 (1762500021) [10284.351062] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10284.734154] LustreError: 386638:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10284.771894] LustreError: 386638:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10284.820213] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10289.286965] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [10290.144909] LustreError: 386638:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10295.283228] LustreError: 386638:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10297.792912] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10298.216570] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10301.897951] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [10307.829842] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10311.799848] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10318.906886] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10319.301404] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10325.000749] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [10330.295392] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10336.265170] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10336.346692] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10336.509370] Lustre: lustre-OST0001: new disk, initializing [10336.513979] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10336.588186] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10341.947166] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [10346.629943] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10346.636767] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10346.720790] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10348.835609] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [10354.233478] LustreError: 389840:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [10354.241441] LustreError: 389840:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [10354.313970] Lustre: server umount lustre-OST0001 complete [10355.643703] LustreError: 388238:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10357.224237] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10357.226443] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10362.338349] LustreError: 388236:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10362.345395] LustreError: 388236:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10371.554207] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10371.562060] LustreError: Skipped 1 previous similar message [10371.565605] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10371.593495] Lustre: Skipped 1 previous similar message [10371.607081] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10372.578248] LustreError: 388236:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10372.603951] LustreError: 388236:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10375.704781] LustreError: 390064:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10375.728772] LustreError: 390064:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10375.919354] Lustre: server umount lustre-OST0000 complete [10380.257307] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10380.263072] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10380.278120] Lustre: Skipped 1 previous similar message [10380.280876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10380.283579] Lustre: Skipped 2 previous similar messages [10385.257836] LustreError: 390265:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [10385.260191] LustreError: 390265:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10385.428211] Lustre: server umount lustre-MDT0000 complete [10388.245484] LustreError: 386618:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762500132 with bad export cookie 10968251548520958829 [10388.252716] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10388.256143] LustreError: 386618:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [10394.646315] Lustre: server umount lustre-MDT0001 complete [10402.876177] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [10405.392022] Key type lgssc unregistered [10405.682708] LNet: 390999:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10405.687758] LNetError: 390999:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10405.705786] LNet: Removed LNI 192.168.201.108@tcp [10406.392979] Key type .llcrypt unregistered [10406.395573] Key type ._llcrypt unregistered [10436.065213] Key type ._llcrypt registered [10436.066909] Key type .llcrypt registered [10436.200747] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_hostid [10450.612126] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [10452.688027] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10452.735144] alg: No test for adler32 (adler32-zlib) [10453.920204] Lustre: Lustre: Build Version: 2.16.58_106_gac70ede [10454.219694] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [10455.896122] Key type lgssc registered [10457.442320] Lustre: Echo OBD driver; http://www.lustre.org/ [10465.992385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10473.445640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10479.928526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10486.124659] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10491.856032] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10491.899767] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10491.913464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10493.084559] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10493.125441] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10493.191388] Lustre: lustre-MDT0000: new disk, initializing [10493.290132] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10493.313518] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10496.385573] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [10504.721667] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10504.771268] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10504.857958] Lustre: 394950:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10504.906918] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10504.909470] Lustre: Skipped 1 previous similar message [10504.961462] Lustre: lustre-MDT0001: new disk, initializing [10505.053958] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10505.086486] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10505.097967] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10508.022753] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [10512.837594] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10515.350651] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10520.203492] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10520.263508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10520.538086] Lustre: lustre-OST0000: new disk, initializing [10520.544359] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10520.627261] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10525.098400] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [10527.288533] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10527.296374] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10527.411561] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10531.816730] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10535.339859] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10535.510653] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10538.548597] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10538.738184] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10543.072182] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10543.075614] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10543.076307] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10543.085151] Lustre: Skipped 1 previous similar message [10543.100096] Lustre: Skipped 1 previous similar message [10546.284768] LustreError: 397890:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [10546.673830] Lustre: server umount lustre-OST0000 complete [10551.266514] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10551.284910] LustreError: Skipped 1 previous similar message [10551.303085] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10551.330823] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10553.314318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10553.321061] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10557.283011] LustreError: 398091:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [10557.286248] LustreError: 398091:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10557.388383] Lustre: server umount lustre-MDT0000 complete [10561.637345] LustreError: 395733:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762500306 with bad export cookie 9033875355263978751 [10561.639204] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10561.723091] LustreError: 395733:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [10561.862482] LustreError: 398294:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10561.866682] LustreError: 398294:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10562.233354] Lustre: server umount lustre-MDT0001 complete [10573.445144] Lustre: DEBUG MARKER: == conf-sanity test 65: re-create the lost last_rcvd file when server mount ========================================================== 02:25:15 (1762500315) [10584.424398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10591.201709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10591.500560] Lustre: lustre-MDT0000: new disk, initializing [10591.530780] LustreError: 399561:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10591.542372] LustreError: 399561:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10591.609895] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10594.924713] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [10596.834578] LustreError: 399561:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10600.932437] LustreError: 399561:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10601.738381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10602.016629] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10605.239554] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [10609.566048] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10613.433659] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10615.447039] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10615.448635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10615.456768] Lustre: Skipped 1 previous similar message [10615.459767] Lustre: Skipped 1 previous similar message [10620.896463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10620.907841] Lustre: Skipped 2 previous similar messages [10621.600987] LustreError: 400906:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [10621.611886] LustreError: 400906:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [10621.660299] Lustre: server umount lustre-MDT0000 complete [10624.621827] LustreError: 399542:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762500369 with bad export cookie 9033875355263979633 [10624.625546] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10624.628331] LustreError: 399542:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [10624.935463] Lustre: server umount lustre-MDT0001 complete [10632.546292] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 02:26:15 (1762500375) [10638.300307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10638.722308] LustreError: 401828:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10638.855042] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10642.468696] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [10643.937992] LustreError: 401828:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10649.199507] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10652.434547] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [10658.167623] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10662.625367] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10669.350947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10669.676677] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10669.683827] Lustre: Skipped 1 previous similar message [10674.101323] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [10679.034275] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10689.621793] Lustre: Permanently deactivating lustre-OST0000 [10689.636362] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [10697.752300] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10699.751633] LustreError: 404521:0:(mgs_llog.c:1937:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [10699.755321] LustreError: 404521:0:(mgs_llog.c:1991:mgs_replace_nids()) Only MGS is allowed to be started [10699.758313] LustreError: 404521:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10701.598143] LustreError: 404569:0:(mgs_llog.c:1991:mgs_replace_nids()) Only MGS is allowed to be started [10701.602509] LustreError: 404569:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10709.107175] LustreError: 404666:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10709.113971] LustreError: 404666:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [10709.242750] Lustre: server umount lustre-OST0000 complete [10711.033674] LustreError: 404818:0:(mgs_llog.c:1991:mgs_replace_nids()) Only MGS is allowed to be started [10711.037489] LustreError: 404818:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10712.545581] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10712.555644] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10712.566588] Lustre: Skipped 1 previous similar message [10721.774092] LustreError: 404916:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [10721.776931] LustreError: 404916:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10721.841362] Lustre: server umount lustre-MDT0000 complete [10722.784369] LustreError: 401828:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10722.797712] LustreError: 401828:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10724.620511] LustreError: 401807:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762500469 with bad export cookie 9033875355263980186 [10724.628716] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10724.653026] LustreError: 401807:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [10735.227741] Lustre: server umount lustre-MDT0001 complete [10741.898218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10745.379038] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [10747.640765] LustreError: 405959:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [10753.417908] Lustre: 406299:0:(mgs_llog.c:1698:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.108@tcp [10755.504567] LustreError: 406397:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [10755.513265] LustreError: 406397:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [10755.610568] Lustre: server umount MGS complete [10764.672777] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10765.128619] LustreError: 407025:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10765.144644] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10765.151025] Lustre: Skipped 1 previous similar message [10765.200099] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10768.565440] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [10775.171347] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10775.355993] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10778.596367] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [10784.300548] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10788.520857] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10790.999063] Lustre: Permanently reactivating lustre-OST0000 [10791.022440] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [10791.025899] Lustre: Skipped 2 previous similar messages [10814.432552] LustreError: 408450:0:(osp_dev.c:692:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [10820.956581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10825.840963] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [10830.670676] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10840.036308] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10840.040885] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10840.042104] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10840.042109] Lustre: Skipped 2 previous similar messages [10840.059538] Lustre: Skipped 2 previous similar messages [10840.066176] LustreError: Skipped 1 previous similar message [10841.471381] LustreError: 410048:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10841.474351] LustreError: 410048:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [10841.654879] Lustre: server umount lustre-OST0000 complete [10847.202487] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10851.637955] Lustre: server umount lustre-MDT0000 complete [10855.399839] LustreError: 407025:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10855.432780] LustreError: 407025:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10855.860908] LustreError: 407004:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762500600 with bad export cookie 9033875355263981145 [10855.868297] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10855.887415] LustreError: 407004:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [10856.264912] Lustre: server umount lustre-MDT0001 complete [10864.962611] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [10867.249479] Key type lgssc unregistered [10867.530180] LNet: 410988:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10867.538260] LNetError: 410988:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10867.555864] LNet: Removed LNI 192.168.201.108@tcp [10868.450357] Key type .llcrypt unregistered [10868.457719] Key type ._llcrypt unregistered [10882.791225] Key type ._llcrypt registered [10882.794384] Key type .llcrypt registered [10882.867826] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_hostid [10894.188930] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [10894.805974] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10894.821232] alg: No test for adler32 (adler32-zlib) [10895.816638] Lustre: Lustre: Build Version: 2.16.58_106_gac70ede [10895.990235] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [10897.631184] Key type lgssc registered [10898.403503] Lustre: Echo OBD driver; http://www.lustre.org/ [10904.503893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10909.286179] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10913.737753] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10919.104046] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10924.972556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10925.034124] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10925.047868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10926.219481] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10926.250197] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10926.297656] Lustre: lustre-MDT0000: new disk, initializing [10926.371341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10926.383230] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10929.139717] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [10938.006799] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10938.075431] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10938.172331] Lustre: 414100:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10938.252338] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10938.256086] Lustre: Skipped 1 previous similar message [10938.302907] Lustre: lustre-MDT0001: new disk, initializing [10938.380798] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10938.429241] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10938.448030] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10941.697150] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [10948.263759] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10951.098784] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10956.806313] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10956.905965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10957.099800] Lustre: lustre-OST0000: new disk, initializing [10957.103598] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10957.170820] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10961.568731] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [10966.076815] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10966.084451] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10966.142319] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10967.194402] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10970.134385] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10970.294123] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10973.284519] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10973.445821] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10976.224684] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10976.228954] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10976.241217] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10976.739128] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10976.746553] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10976.757325] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10981.054640] LustreError: 417038:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [10981.265538] Lustre: server umount lustre-OST0000 complete [10984.418913] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10984.427939] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10984.444575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10986.976861] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10986.977097] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10990.367619] LustreError: 417239:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [10990.370251] LustreError: 417239:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10990.451328] Lustre: server umount lustre-MDT0000 complete [10993.218098] LustreError: 414093:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762500737 with bad export cookie 14881281783730374085 [10993.226717] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10993.329264] LustreError: 417441:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10993.332402] LustreError: 417441:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10993.522706] Lustre: server umount lustre-MDT0001 complete [11000.219619] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11001.607044] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 02:32:24 (1762500744) [11006.498089] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11006.839133] LustreError: 418163:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11006.859183] LustreError: 418163:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11006.929413] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11010.049258] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [11012.066705] LustreError: 418163:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11016.608600] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11016.910970] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11019.558775] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [11023.114928] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11025.952725] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11031.751637] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11032.243645] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11037.790500] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [11042.390451] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11044.309651] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11046.863863] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11053.032055] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11053.037065] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11053.045175] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11053.049049] Lustre: Skipped 1 previous similar message [11057.326401] LustreError: 420600:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [11057.330562] LustreError: 420600:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [11057.475041] Lustre: server umount lustre-OST0000 complete [11063.264212] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11063.265789] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11063.272307] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11063.280289] Lustre: Skipped 2 previous similar messages [11063.287664] Lustre: Skipped 2 previous similar messages [11066.858728] LustreError: 420802:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [11066.869770] LustreError: 420802:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11066.957299] Lustre: server umount lustre-MDT0000 complete [11068.389375] LustreError: 418162:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11068.403204] LustreError: 418162:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11070.187981] LustreError: 418143:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762500814 with bad export cookie 14881281783730374967 [11070.211309] LustreError: 418143:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [11070.231475] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11070.522139] Lustre: server umount lustre-MDT0001 complete [11078.783591] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [11081.162347] Key type lgssc unregistered [11081.431105] LNet: 421535:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11081.439363] LNetError: 421535:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11081.455360] LNet: Removed LNI 192.168.201.108@tcp [11082.310879] Key type .llcrypt unregistered [11082.319643] Key type ._llcrypt unregistered [11101.742997] Key type ._llcrypt registered [11101.744940] Key type .llcrypt registered [11101.813459] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [11102.082313] Key type .llcrypt unregistered [11102.093316] Key type ._llcrypt unregistered [11116.724530] Key type ._llcrypt registered [11116.726711] Key type .llcrypt registered [11116.823432] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [11117.827891] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11117.851991] alg: No test for adler32 (adler32-zlib) [11118.850397] Lustre: Lustre: Build Version: 2.16.58_106_gac70ede [11119.035152] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [11120.679182] Key type lgssc registered [11121.488709] Lustre: Echo OBD driver; http://www.lustre.org/ [11126.905283] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11126.920228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11128.256468] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11131.311480] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [11138.071220] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11138.318540] LustreError: 423708:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11138.331881] LustreError: 423708:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11138.355377] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11139.365142] LustreError: 423708:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11139.376209] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11139.389133] LustreError: 423708:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11142.983450] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [11144.676248] LustreError: 423708:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11144.692939] LustreError: 423708:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11147.313867] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11149.793067] LustreError: 423708:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11149.818322] LustreError: 423708:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11152.787508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11153.059851] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11156.202488] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [11168.224869] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11168.235184] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11168.250992] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11168.748971] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11169.095076] LustreError: 425761:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [11169.282935] Lustre: server umount lustre-OST0000 complete [11173.344172] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11173.346877] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11173.378530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11173.382147] Lustre: Skipped 1 previous similar message [11178.976087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11178.984760] Lustre: Skipped 1 previous similar message [11179.280365] LustreError: 425963:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [11179.285165] LustreError: 425963:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11179.384736] Lustre: server umount lustre-MDT0000 complete [11182.831266] LustreError: 423688:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762500927 with bad export cookie 6830262023002136390 [11182.862426] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11183.001438] LustreError: 426164:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [11183.005198] LustreError: 426164:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11183.194146] Lustre: server umount lustre-MDT0001 complete [11192.588696] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [11195.551053] Key type lgssc unregistered [11195.852331] LNet: 426695:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11195.858200] LNetError: 426695:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11195.874799] LNet: Removed LNI 192.168.201.108@tcp [11196.534209] Key type .llcrypt unregistered [11196.537187] Key type ._llcrypt unregistered [11216.932693] Key type ._llcrypt registered [11216.933973] Key type .llcrypt registered [11217.025935] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [11217.948491] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11218.033096] alg: No test for adler32 (adler32-zlib) [11219.218919] Lustre: Lustre: Build Version: 2.16.58_106_gac70ede [11219.455464] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [11221.111174] Key type lgssc registered [11222.323284] Lustre: Echo OBD driver; http://www.lustre.org/ [11230.044030] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11230.095833] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11245.727413] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [11250.847219] LustreError: 428063:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.108@tcp: failed processing log, type 4: rc = -110 [11277.599868] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11282.671764] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [11286.904824] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11292.647337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11293.076512] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11293.153780] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11297.407517] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [11299.299020] LustreError: 428986:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11299.359568] LustreError: 428986:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [11304.419394] LustreError: 428089:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11304.432310] LustreError: 428089:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11305.816851] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11306.240557] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11310.374801] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [11316.166726] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11319.653630] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11326.947537] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11326.953801] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11326.956188] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11330.595366] LustreError: 430344:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [11330.745776] Lustre: server umount lustre-OST0000 complete [11334.114511] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [11334.135664] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11334.156152] Lustre: Skipped 1 previous similar message [11334.161164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11334.164098] Lustre: Skipped 1 previous similar message [11334.323263] LustreError: 430547:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [11334.333621] LustreError: 430547:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11334.458709] Lustre: server umount lustre-MDT0000 complete [11337.184503] LustreError: 428992:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11337.186742] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11338.056298] LustreError: 428071:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762501082 with bad export cookie 14061255475520479888 [11338.062129] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11338.068656] LustreError: 428071:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [11338.180555] LustreError: 430748:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [11338.188313] LustreError: 430748:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11338.400204] Lustre: server umount lustre-MDT0001 complete [11348.709870] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [11351.382482] Key type lgssc unregistered [11351.654421] LNet: 431280:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11351.663702] LNetError: 431280:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11352.681885] LNet: Removed LNI 192.168.201.108@tcp [11353.532634] Key type .llcrypt unregistered [11353.535195] Key type ._llcrypt unregistered [11373.820995] Key type ._llcrypt registered [11373.823052] Key type .llcrypt registered [11373.916794] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [11374.699179] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11374.713688] alg: No test for adler32 (adler32-zlib) [11375.726386] Lustre: Lustre: Build Version: 2.16.58_106_gac70ede [11375.958087] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [11377.583202] Key type lgssc registered [11378.533748] Lustre: Echo OBD driver; http://www.lustre.org/ [11385.192269] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11385.215144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11386.622431] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11389.784631] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [11397.093125] LustreError: 432657:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11397.115915] LustreError: 432657:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11397.191438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11397.487495] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11400.755463] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [11405.236247] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11408.347016] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11414.450212] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11414.767255] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11420.654031] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [11426.576711] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11432.186345] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11437.544832] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11437.547386] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11437.563304] Lustre: Skipped 2 previous similar messages [11437.567661] Lustre: Skipped 1 previous similar message [11441.023794] LustreError: 434901:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [11441.182505] Lustre: server umount lustre-MDT0000 complete [11442.657914] LustreError: 433038:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11443.397949] LustreError: 432635:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.8@tcp arrived at 1762501187 with bad export cookie 490586760866430601 [11443.413917] LustreError: 432635:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [11445.214247] LustreError: 432635:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762501189 with bad export cookie 490586760866429929 [11445.226248] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11445.239340] LustreError: 432635:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [11447.776066] LustreError: 432656:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11447.782133] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11447.812168] LustreError: 432656:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11447.839450] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11447.839458] Lustre: Skipped 2 previous similar messages [11447.844674] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11450.785110] LustreError: 432656:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11450.791525] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11450.804251] LustreError: 432656:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11451.314107] LustreError: 435104:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [11451.316638] LustreError: 435104:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11451.459688] Lustre: server umount lustre-OST0000 complete [11454.888751] LustreError: 432656:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11454.907461] LustreError: 432656:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11457.185127] LustreError: 435458:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [11457.188244] LustreError: 435458:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11457.470020] Lustre: server umount lustre-MDT0001 complete [11466.571438] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [11469.152605] Key type lgssc unregistered [11469.455355] LNet: 435989:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11469.466334] LNetError: 435989:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11470.502172] LNet: Removed LNI 192.168.201.108@tcp [11471.291876] Key type .llcrypt unregistered [11471.293715] Key type ._llcrypt unregistered [11492.316859] Key type ._llcrypt registered [11492.322195] Key type .llcrypt registered [11492.468709] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [11494.300536] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11494.369299] alg: No test for adler32 (adler32-zlib) [11495.434646] Lustre: Lustre: Build Version: 2.16.58_106_gac70ede [11495.614200] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [11497.255723] Key type lgssc registered [11498.451612] Lustre: Echo OBD driver; http://www.lustre.org/ [11506.758863] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11506.819923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11508.501237] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11512.877560] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [11517.934719] LustreError: 437388:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11517.940573] LustreError: 437388:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11523.045390] LustreError: 437388:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11523.456633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11523.877330] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11527.387367] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [11532.513334] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11536.080707] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11541.911758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11542.282548] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11547.015988] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [11551.853623] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11553.578280] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11559.905799] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [11559.911173] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11559.925691] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11562.929371] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.8@tcp (stopping) [11564.009421] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11564.013034] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11565.719151] LustreError: 439634:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [11565.814156] Lustre: server umount lustre-MDT0001 complete [11569.124569] LustreError: 437388:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11569.157865] LustreError: 437388:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11574.242050] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11574.247087] LustreError: 437388:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11574.256117] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11574.264031] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11574.349546] Lustre: Skipped 1 previous similar message [11577.978549] LustreError: 439836:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [11577.986343] LustreError: 439836:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11578.144988] Lustre: server umount lustre-OST0000 complete [11579.360996] LustreError: 437388:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11581.594469] LustreError: 440039:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [11581.597815] LustreError: 440039:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11581.795795] Lustre: server umount lustre-MDT0000 complete [11591.491856] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [11593.842349] Key type lgssc unregistered [11594.153866] LNet: 440723:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11594.168289] LNetError: 440723:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11594.189595] LNet: Removed LNI 192.168.201.108@tcp [11594.835539] Key type .llcrypt unregistered [11594.837307] Key type ._llcrypt unregistered [11613.122440] Key type ._llcrypt registered [11613.125494] Key type .llcrypt registered [11613.201439] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_hostid [11624.157817] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [11624.718651] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11624.745191] alg: No test for adler32 (adler32-zlib) [11625.764941] Lustre: Lustre: Build Version: 2.16.58_106_gac70ede [11625.956853] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [11627.607730] Key type lgssc registered [11628.478119] Lustre: Echo OBD driver; http://www.lustre.org/ [11635.426993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11640.020169] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11644.128283] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11649.040831] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11658.478630] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [11667.204668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11672.983442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11673.041886] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11673.062765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11674.267387] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11674.326860] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11674.383995] Lustre: lustre-MDT0000: new disk, initializing [11674.507705] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11674.538344] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11678.351558] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [11684.944918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11690.007697] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11690.074144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11690.419666] Lustre: lustre-OST0000: new disk, initializing [11690.428683] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11690.435075] Lustre: Skipped 1 previous similar message [11690.490243] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11695.241081] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [11698.757456] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [11698.763870] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [11698.795990] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [11701.144185] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11707.772174] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11713.094499] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11713.181649] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11713.318712] Lustre: 444678:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11713.403642] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11713.456606] Lustre: lustre-MDT0001: new disk, initializing [11713.588234] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11713.628237] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [11713.635792] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [11716.868858] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [11724.820440] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [11725.126739] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [11729.991754] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [11730.218468] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [11738.587324] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.8@tcp (stopping) [11740.139574] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11740.153436] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11740.177501] Lustre: Skipped 1 previous similar message [11740.202743] Lustre: Skipped 1 previous similar message [11741.837464] LustreError: 448096:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [11741.974718] Lustre: server umount lustre-MDT0001 complete [11750.375508] LustreError: 444684:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11750.396997] LustreError: 444684:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [11755.494869] LustreError: 444685:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11755.528655] LustreError: 444685:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11760.096480] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11760.108375] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11760.131375] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11760.614502] LustreError: 447276:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11765.729554] LustreError: 447276:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11765.729655] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11766.012174] LustreError: 448369:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [11766.015057] LustreError: 448369:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11766.258640] Lustre: server umount lustre-OST0000 complete [11775.832501] LustreError: 448570:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [11775.850612] LustreError: 448570:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11776.103938] Lustre: server umount lustre-MDT0000 complete [11788.820556] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [11791.633329] Key type lgssc unregistered [11791.971800] LNet: 449255:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11791.984870] LNetError: 449255:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11792.003879] LNet: Removed LNI 192.168.201.108@tcp [11792.802842] Key type .llcrypt unregistered [11792.806657] Key type ._llcrypt unregistered [11826.761365] Key type ._llcrypt registered [11826.762818] Key type .llcrypt registered [11826.851867] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [11827.088529] Key type .llcrypt unregistered [11827.094240] Key type ._llcrypt unregistered [11839.918264] Key type ._llcrypt registered [11839.920540] Key type .llcrypt registered [11840.002824] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [11840.676758] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11840.729960] alg: No test for adler32 (adler32-zlib) [11841.778783] Lustre: Lustre: Build Version: 2.16.58_106_gac70ede [11842.039143] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [11843.679137] Key type lgssc registered [11844.632557] Lustre: Echo OBD driver; http://www.lustre.org/ [11851.040478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11856.327882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11861.640711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11867.053545] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11867.099774] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11867.116619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11868.321954] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11868.367632] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11868.430557] Lustre: lustre-MDT0000: new disk, initializing [11868.509245] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11868.529588] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11871.258345] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [11879.232425] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11879.279437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11879.350862] Lustre: 452582:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11879.398132] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11879.401811] Lustre: Skipped 1 previous similar message [11879.455545] Lustre: lustre-MDT0001: new disk, initializing [11879.538462] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11879.591059] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11879.613114] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11882.618448] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [11889.254391] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11892.444854] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11898.147725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11898.214294] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11898.450853] Lustre: lustre-OST0000: new disk, initializing [11898.455758] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11898.544773] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11903.240566] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [11904.051230] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11904.061236] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11904.165343] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11909.943697] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11915.238461] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11915.248195] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11915.256584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11919.846899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11919.850863] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11921.148226] LustreError: 455025:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [11921.290868] Lustre: server umount lustre-MDT0000 complete [11924.511287] LustreError: 452574:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762501668 with bad export cookie 18289630706282152106 [11924.518051] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11924.535161] LustreError: 452574:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [11924.724334] LustreError: 455228:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [11924.727272] LustreError: 455228:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11924.943984] Lustre: server umount lustre-MDT0001 complete [11938.490787] LustreError: 455429:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [11938.494747] LustreError: 455429:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11938.623617] Lustre: server umount lustre-OST0000 complete [11946.854959] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 02:48:09 (1762501689) [11947.705511] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11953.266700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11953.777099] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11957.884556] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [11958.753486] LustreError: 456258:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11958.781128] LustreError: 456258:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11963.874289] LustreError: 456258:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11966.217449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11966.567539] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11970.151881] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [11975.129278] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11978.144183] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11983.885748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11983.951170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11984.076136] Lustre: Found index 0 for lustre-OST0000, updating log [11984.260656] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11989.533563] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [11994.429466] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11999.624215] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [12004.833402] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12004.839038] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12004.842291] Lustre: Skipped 2 previous similar messages [12004.850753] Lustre: Skipped 2 previous similar messages [12005.111073] LustreError: 458514:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [12005.114394] LustreError: 458514:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [12005.242627] Lustre: server umount lustre-OST0000 complete [12007.694665] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12010.976126] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12010.978083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12010.980926] Lustre: Skipped 1 previous similar message [12010.987074] Lustre: Skipped 1 previous similar message [12015.375323] LustreError: 458768:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [12015.378406] LustreError: 458768:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [12015.464875] Lustre: server umount lustre-MDT0000 complete [12016.095888] LustreError: 456258:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12016.108209] LustreError: 456258:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12017.867185] LustreError: 456237:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762501762 with bad export cookie 18289630706282153499 [12017.874824] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12017.887719] LustreError: 456237:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [12018.188606] Lustre: server umount lustre-MDT0001 complete [12025.195718] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 02:49:27 (1762501767) [12026.297150] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12027.281653] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12028.228055] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12029.122575] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12030.038896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12030.737050] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12031.549854] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12032.223789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12032.856487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12033.509129] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12034.244292] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12035.107375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12035.923181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12036.662380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12037.366682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12038.114907] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12038.971205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12039.732764] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12040.459960] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12041.104740] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12041.823804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12042.509715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12043.191584] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12043.873267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12044.617784] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12045.357381] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12046.110935] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12046.905599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12047.699567] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12048.649077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12049.537429] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12050.417218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12051.330519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12052.081602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12052.799787] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12053.592546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12054.483704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12055.236628] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12056.130793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12057.000893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12057.795830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12058.467168] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12059.219961] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12060.032333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12060.863716] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12061.770278] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12062.787040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12063.658346] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12064.480840] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12065.239209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12066.084319] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12071.929121] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12072.235291] LustreError: 462367:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12072.283642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12074.851129] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [12077.537025] LustreError: 462367:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12080.988910] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12084.498950] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [12090.010402] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12093.762902] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12102.015473] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12102.182614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12102.733395] Lustre: Found index 0 for lustre-OST0000, updating log [12110.242747] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [12116.802254] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12118.640213] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [12124.133328] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12124.133586] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12124.137062] LustreError: Skipped 1 previous similar message [12124.143485] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12124.164959] Lustre: Skipped 1 previous similar message [12133.343608] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12133.377803] LustreError: 464631:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [12133.385989] LustreError: 464631:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [12133.523284] Lustre: server umount lustre-OST0000 complete [12137.440573] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12137.444124] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12137.449081] Lustre: Skipped 1 previous similar message [12137.451484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12151.263168] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12151.388406] LustreError: 464836:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [12151.399362] LustreError: 464836:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [12151.484185] Lustre: server umount lustre-MDT0000 complete [12151.788693] LustreError: 462362:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12151.800803] LustreError: 462362:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12155.386571] LustreError: 462348:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762501899 with bad export cookie 18289630706282154584 [12155.398477] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12155.403574] LustreError: 462348:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [12155.743341] Lustre: server umount lustre-MDT0001 complete [12158.711209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12173.177433] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_hostid [12182.123995] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [12191.046990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12196.837394] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12202.202940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12207.370772] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12212.828369] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 02:52:35 (1762501955) [12226.684304] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [12228.651531] Key type lgssc unregistered [12228.880868] LNet: 468609:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12228.893334] LNetError: 468609:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [12354.852639] LNet: Removed LNI 192.168.201.108@tcp [12355.667524] Key type .llcrypt unregistered [12355.669134] Key type ._llcrypt unregistered [12367.163627] Key type ._llcrypt registered [12367.165548] Key type .llcrypt registered [12367.243018] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [12367.897740] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12367.949461] alg: No test for adler32 (adler32-zlib) [12368.944290] Lustre: Lustre: Build Version: 2.16.58_106_gac70ede [12369.129653] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [12370.775175] Key type lgssc registered [12371.631276] Lustre: Echo OBD driver; http://www.lustre.org/ [12375.018274] LNet: Added LNI 192.168.201.108@tcp100 [8/256/0/180] [12376.263373] Lustre: DEBUG MARKER: SKIP: conf-sanity test_73c unable to configure net [12388.140374] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [12390.343297] Key type lgssc unregistered [12390.637584] LNet: 470477:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12390.642143] LNetError: 470477:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [12390.647943] LNet: Removed LNI 192.168.201.108@tcp [12390.654561] LNet: Removed LNI 192.168.201.108@tcp100 [12391.157343] Key type .llcrypt unregistered [12391.158943] Key type ._llcrypt unregistered [12407.236456] Key type ._llcrypt registered [12407.238045] Key type .llcrypt registered [12407.315270] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_hostid [12417.547421] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [12418.081743] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12418.100205] alg: No test for adler32 (adler32-zlib) [12419.075975] Lustre: Lustre: Build Version: 2.16.58_106_gac70ede [12419.255392] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [12420.895889] Key type lgssc registered [12421.673844] Lustre: Echo OBD driver; http://www.lustre.org/ [12428.391286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12433.008409] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12437.158038] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12442.143322] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12448.215407] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 02:56:30 (1762502190) [12461.974947] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [12464.012796] Key type lgssc unregistered [12464.264267] LNet: 474493:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12464.269902] LNetError: 474493:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [12464.287565] LNet: Removed LNI 192.168.201.108@tcp [12464.960712] Key type .llcrypt unregistered [12464.962309] Key type ._llcrypt unregistered [12478.191373] Key type ._llcrypt registered [12478.194069] Key type .llcrypt registered [12478.287449] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [12478.957447] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12479.055815] alg: No test for adler32 (adler32-zlib) [12480.063094] Lustre: Lustre: Build Version: 2.16.58_106_gac70ede [12480.248714] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [12481.871365] Key type lgssc registered [12482.578900] Lustre: Echo OBD driver; http://www.lustre.org/ [12489.136890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12489.181603] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12489.200746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12490.368227] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12490.399290] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12490.458903] Lustre: lustre-MDT0000: new disk, initializing [12490.527707] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12490.541339] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12493.041409] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [12500.345814] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12500.408652] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12500.503685] Lustre: 475684:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12500.556189] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12500.559094] Lustre: Skipped 1 previous similar message [12500.617503] Lustre: lustre-MDT0001: new disk, initializing [12500.689503] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12500.727239] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12500.734570] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12503.425367] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [12508.579835] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12510.985182] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12515.457584] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12515.492086] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12515.650262] Lustre: lustre-OST0000: new disk, initializing [12515.660605] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12515.722362] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12519.315737] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [12521.500308] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12521.506977] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12521.592531] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12524.373524] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12526.497556] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing lnet_if_list [12529.504555] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [12529.523435] LNet: Added LNI 192.168.201.108@tcp42 [8/256/0/180] [12541.423117] LustreError: 479065:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [12541.522976] Lustre: server umount lustre-OST0000 complete [12547.042442] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12547.044395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12547.046842] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12547.047688] Lustre: Skipped 1 previous similar message [12547.071686] Lustre: Skipped 1 previous similar message [12552.160624] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12557.280707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12557.285759] Lustre: Skipped 2 previous similar messages [12558.303187] Lustre: 474955:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1762502286/real 1762502286] req@00000000faf58b59 x1848117532067200/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1762502302 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12558.303495] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12558.303601] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12558.322038] Lustre: 474955:0:(client.c:2468:ptlrpc_expire_one_request()) Skipped 1 previous similar message [12558.409728] LustreError: 479266:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [12558.415717] LustreError: 479266:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [12558.474414] Lustre: server umount lustre-MDT0000 complete [12560.549589] LustreError: 475675:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762502304 with bad export cookie 5070072593219363589 [12560.552598] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12560.556677] LustreError: 475675:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [12560.651664] LustreError: 479468:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [12560.653866] LustreError: 479468:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [12560.802948] Lustre: server umount lustre-MDT0001 complete [12567.807338] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [12569.231762] Key type lgssc unregistered [12569.437382] LNet: 480009:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12569.442028] LNetError: 480009:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [12569.447455] LNet: Removed LNI 192.168.201.108@tcp [12569.455231] LNet: Removed LNI 192.168.201.108@tcp42 [12569.905656] Key type .llcrypt unregistered [12569.907203] Key type ._llcrypt unregistered [12582.227724] Key type ._llcrypt registered [12582.229426] Key type .llcrypt registered [12582.313362] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [12582.888792] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12582.932087] alg: No test for adler32 (adler32-zlib) [12583.914053] Lustre: Lustre: Build Version: 2.16.58_106_gac70ede [12584.132975] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [12585.791151] Key type lgssc registered [12586.790460] Lustre: Echo OBD driver; http://www.lustre.org/ [12592.860023] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12592.885946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12594.162290] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12596.761367] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [12602.961794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12603.195260] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12606.251581] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [12610.346376] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12613.339836] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12618.445282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12618.768560] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12622.766945] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [12627.072155] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12635.094646] loop0: detected capacity change from 0 to 204800000 [12635.279324] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12635.291520] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12635.299070] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12635.310647] blk_update_request: operation not supported error, dev loop0, sector 160240 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12635.383887] blk_update_request: operation not supported error, dev loop0, sector 160344 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12635.437108] blk_update_request: operation not supported error, dev loop0, sector 168552 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12635.477821] blk_update_request: operation not supported error, dev loop0, sector 173280 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12635.538925] blk_update_request: operation not supported error, dev loop0, sector 176912 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12635.597314] blk_update_request: operation not supported error, dev loop0, sector 185112 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12635.645547] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12640.087996] loop0: detected capacity change from 0 to 204800000 [12640.268553] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12640.400193] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12643.932151] loop0: detected capacity change from 0 to 204800000 [12647.653213] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12647.721468] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12647.809519] Lustre: fs15246-OST0000: new disk, initializing [12647.817181] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [12647.864894] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12651.771760] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [12658.483831] loop1: detected capacity change from 0 to 204800000 [12662.175705] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12662.215146] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12662.261905] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [12662.301654] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [12662.348255] Lustre: fs15246-MDT0000: new disk, initializing [12662.446124] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12662.465701] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12662.539310] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [12663.583995] Lustre: 485604:0:(client.c:1627:after_reply()) @@@ resending request on EINPROGRESS req@00000000aa85935b x1848117641123072/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1762502423 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [12664.676587] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [12671.657050] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12671.664444] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12671.676177] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12680.169685] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12680.179231] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [12685.209676] LustreError: 486731:0:(obd_class.h:479:obd_check_dev()) Device 30 not setup [12685.283726] Lustre: server umount fs15246-MDT0000 complete [12685.286436] LustreError: 481255:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12685.306330] LustreError: 481255:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12690.409897] LustreError: 481227:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12691.192619] LustreError: 487235:0:(obd_class.h:479:obd_check_dev()) Device 32 not setup [12691.194528] LustreError: 487235:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [12691.229290] Lustre: server umount fs15246-OST0000 complete [12699.616949] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12699.620523] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12699.627882] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12700.645979] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12700.654209] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12704.507112] LustreError: 487736:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [12704.509747] LustreError: 487736:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [12704.667099] Lustre: server umount lustre-OST0000 complete [12710.881395] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12710.883513] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12710.894792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12710.895960] Lustre: Skipped 1 previous similar message [12710.906240] Lustre: Skipped 2 previous similar messages [12713.583820] LustreError: 487938:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [12713.594744] LustreError: 487938:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [12713.654191] Lustre: server umount lustre-MDT0000 complete [12716.004842] LustreError: 481225:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12716.025927] LustreError: 481225:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12716.585057] LustreError: 481212:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762502460 with bad export cookie 7939112740247769564 [12716.587694] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12716.594349] LustreError: 481212:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [12716.990087] Lustre: server umount lustre-MDT0001 complete [12725.068583] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [12726.842134] Key type lgssc unregistered [12727.079084] LNet: 488682:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12727.085286] LNetError: 488682:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [12728.103489] LNet: Removed LNI 192.168.201.108@tcp [12728.691088] Key type .llcrypt unregistered [12728.692796] Key type ._llcrypt unregistered [12741.840272] Key type ._llcrypt registered [12741.842034] Key type .llcrypt registered [12741.954551] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [12742.694299] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12742.713946] alg: No test for adler32 (adler32-zlib) [12743.697459] Lustre: Lustre: Build Version: 2.16.58_106_gac70ede [12743.872353] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [12745.495611] Key type lgssc registered [12746.275590] Lustre: Echo OBD driver; http://www.lustre.org/ [12751.838602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12755.363502] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12759.123659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12762.325812] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12772.810837] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_hostid [12779.053832] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [12786.281737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12791.242762] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12795.889444] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12800.520109] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12805.231293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12805.284670] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12805.301464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12806.529140] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12806.559912] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12806.628981] Lustre: lustre-MDT0000: new disk, initializing [12806.715126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12806.747165] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12809.410611] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [12816.230587] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12816.277857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12816.346940] Lustre: 493093:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12816.393845] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12816.400062] Lustre: Skipped 1 previous similar message [12816.462965] Lustre: lustre-MDT0001: new disk, initializing [12816.561236] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12816.589977] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12816.597065] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12819.014239] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [12824.311696] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12826.572442] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12831.052662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12831.103362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12831.265215] Lustre: lustre-OST0000: new disk, initializing [12831.271239] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12831.334525] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12835.137589] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [12838.950665] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12838.955403] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12839.069269] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12840.238752] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12843.032716] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12843.154878] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12845.382297] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12845.510717] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12849.121663] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12849.125903] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12849.132690] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12849.634505] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12849.634562] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12849.644944] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12852.602396] LustreError: 496030:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [12852.719562] Lustre: server umount lustre-OST0000 complete [12857.825819] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12857.831590] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12857.840323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12859.873094] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12859.877529] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12861.011360] LustreError: 496233:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [12861.017586] LustreError: 496233:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [12861.061180] Lustre: server umount lustre-MDT0000 complete [12863.087535] LustreError: 496031:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762502607 with bad export cookie 1374262638427436277 [12863.090876] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12863.093915] LustreError: 496031:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [12863.183831] LustreError: 496435:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [12863.186701] LustreError: 496435:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [12863.310842] Lustre: server umount lustre-MDT0001 complete [12877.872754] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_hostid [12883.960685] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [12890.600878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12895.872415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12899.605679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12903.693256] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12904.706121] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12659 sec ========= 03:04:07 (1762502647) [12905.735329] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 03:04:08 (1762502648) === [12907.992895] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 03:04:11 (1762502651) === [12933.477610] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [12935.312125] Key type lgssc unregistered [12935.553669] LNet: 501289:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12935.559689] LNetError: 501289:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [12935.568734] LNet: Removed LNI 192.168.201.108@tcp [12936.147069] Key type .llcrypt unregistered [12936.148562] Key type ._llcrypt unregistered