[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 478072678 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003360] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.008831] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010012] pid_max: default: 32768 minimum: 301 [ 0.012138] LSM: Security Framework initializing [ 0.013049] Yama: becoming mindful. [ 0.014025] SELinux: Initializing. [ 0.015043] *** VALIDATE selinux *** [ 0.022609] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026398] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028087] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029098] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030105] *** VALIDATE tmpfs *** [ 0.032010] *** VALIDATE proc *** [ 0.032874] *** VALIDATE cgroup *** [ 0.033007] *** VALIDATE cgroup2 *** [ 0.034191] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035153] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037003] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038026] Spectre V2 : User space: Vulnerable [ 0.039013] Speculative Store Bypass: Vulnerable [ 0.042577] debug: unmapping init [mem 0xffffffff96459000-0xffffffff96460fff] [ 0.044246] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045986] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047025] ... version: 2 [ 0.048013] ... bit width: 48 [ 0.049013] ... generic registers: 4 [ 0.050010] ... value mask: 0000ffffffffffff [ 0.051015] ... max period: 00007fffffffffff [ 0.052012] ... fixed-purpose events: 3 [ 0.053011] ... event mask: 000000070000000f [ 0.054313] rcu: Hierarchical SRCU implementation. [ 0.056438] smp: Bringing up secondary CPUs ... [ 0.057390] x86: Booting SMP configuration: [ 0.058018] .... node #0, CPUs: #1 #2 #3 [ 0.064082] smp: Brought up 1 node, 4 CPUs [ 0.066009] smpboot: Max logical packages: 1 [ 0.067013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.155813] node 0 deferred pages initialised in 86ms [ 0.158006] devtmpfs: initialized [ 0.159030] x86/mm: Memory block size: 128MB [ 0.161048] gcov: version magic: 0x41383552 [ 0.162570] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.163082] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.164245] pinctrl core: initialized pinctrl subsystem [ 0.165203] [ 0.165768] ************************************************************* [ 0.166014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.167014] ** ** [ 0.168012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.169012] ** ** [ 0.170010] ** This means that this kernel is built to expose internal ** [ 0.171008] ** IOMMU data structures, which may compromise security on ** [ 0.172010] ** your system. ** [ 0.173012] ** ** [ 0.174015] ** If you see this message and you are not debugging the ** [ 0.175016] ** kernel, report this immediately to your vendor! ** [ 0.176013] ** ** [ 0.177014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.178015] ************************************************************* [ 0.179749] NET: Registered protocol family 16 [ 0.180463] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.181050] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.182051] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.183382] cpuidle: using governor menu [ 0.185463] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.186741] PCI: Using configuration type 1 for base access [ 0.189152] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.200085] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.201012] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.203070] cryptd: max_cpu_qlen set to 1000 [ 0.204207] ACPI: Added _OSI(Module Device) [ 0.205007] ACPI: Added _OSI(Processor Device) [ 0.205900] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.206008] ACPI: Added _OSI(Processor Aggregator Device) [ 0.209054] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.212246] ACPI: Interpreter enabled [ 0.213063] ACPI: PM: (supports S0 S3 S4 S5) [ 0.214029] ACPI: Using IOAPIC for interrupt routing [ 0.215161] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.216384] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.227201] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.230042] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.233022] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.237589] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.243470] acpiphp: Slot [2] registered [ 0.245149] acpiphp: Slot [5] registered [ 0.246119] acpiphp: Slot [6] registered [ 0.247081] acpiphp: Slot [7] registered [ 0.248123] acpiphp: Slot [8] registered [ 0.250136] acpiphp: Slot [9] registered [ 0.251079] acpiphp: Slot [10] registered [ 0.252061] acpiphp: Slot [3] registered [ 0.252987] acpiphp: Slot [4] registered [ 0.255121] acpiphp: Slot [11] registered [ 0.256158] acpiphp: Slot [12] registered [ 0.257166] acpiphp: Slot [13] registered [ 0.259189] acpiphp: Slot [14] registered [ 0.261158] acpiphp: Slot [15] registered [ 0.263144] acpiphp: Slot [16] registered [ 0.264000] acpiphp: Slot [17] registered [ 0.264000] acpiphp: Slot [18] registered [ 0.266202] acpiphp: Slot [19] registered [ 0.267111] acpiphp: Slot [20] registered [ 0.268119] acpiphp: Slot [21] registered [ 0.270199] acpiphp: Slot [22] registered [ 0.272148] acpiphp: Slot [23] registered [ 0.274153] acpiphp: Slot [24] registered [ 0.275182] acpiphp: Slot [25] registered [ 0.277148] acpiphp: Slot [26] registered [ 0.279311] acpiphp: Slot [27] registered [ 0.281149] acpiphp: Slot [28] registered [ 0.283118] acpiphp: Slot [29] registered [ 0.285130] acpiphp: Slot [30] registered [ 0.286113] acpiphp: Slot [31] registered [ 0.288262] PCI host bridge to bus 0000:00 [ 0.290022] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.295040] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.299027] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.302028] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.305031] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.310050] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.311307] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.314000] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.318693] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.329777] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.334166] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.337031] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.339033] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.343048] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.345749] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.348848] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.352359] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.355990] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.361021] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.373027] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.378023] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.390240] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.405019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.415030] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.441018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.456011] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.462015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.471018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.488015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.497947] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.505022] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.515016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.532015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.544811] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.551018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.558017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.584026] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.598809] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.608017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.614020] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.637020] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.649000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.660022] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.669016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.687022] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.700177] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.702566] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.706325] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.709427] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.713212] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.718062] iommu: Default domain type: Passthrough [ 0.720551] SCSI subsystem initialized [ 0.722286] ACPI: bus type USB registered [ 0.724231] usbcore: registered new interface driver usbfs [ 0.727142] usbcore: registered new interface driver hub [ 0.728073] usbcore: registered new device driver usb [ 0.730168] pps_core: LinuxPPS API ver. 1 registered [ 0.731011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.735066] PTP clock support registered [ 0.737092] EDAC MC: Ver: 3.0.0 [ 0.738412] PCI: Using ACPI for IRQ routing [ 0.739000] NetLabel: Initializing [ 0.740015] NetLabel: domain hash size = 128 [ 0.742021] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.744091] NetLabel: unlabeled traffic allowed by default [ 0.747150] vgaarb: loaded [ 0.749342] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.751014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.756435] clocksource: Switched to clocksource kvm-clock [ 0.859279] VFS: Disk quotas dquot_6.6.0 [ 0.860880] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.863137] *** VALIDATE ramfs *** [ 0.864312] *** VALIDATE hugetlbfs *** [ 0.865721] pnp: PnP ACPI init [ 0.867905] pnp: PnP ACPI: found 6 devices [ 0.885168] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.886965] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.888067] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.889265] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.890614] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.891854] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.893313] NET: Registered protocol family 2 [ 0.895305] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.898575] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.900708] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.906416] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.910265] TCP: Hash tables configured (established 65536 bind 65536) [ 0.913848] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.917144] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.919702] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.921972] NET: Registered protocol family 1 [ 0.924453] RPC: Registered named UNIX socket transport module. [ 0.925661] RPC: Registered udp transport module. [ 0.927096] RPC: Registered tcp transport module. [ 0.928408] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.930399] NET: Registered protocol family 44 [ 0.932184] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.933965] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.935319] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.937156] PCI: CLS 0 bytes, default 64 [ 0.938222] Unpacking initramfs... [ 2.252652] debug: unmapping init [mem 0xffff983f7cc54000-0xffff983f7ffbffff] [ 2.256545] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.259068] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.262298] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.764975] Initialise system trusted keyrings [ 2.766766] Key type blacklist registered [ 2.768757] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.777919] zbud: loaded [ 2.780788] *** VALIDATE nfs *** [ 2.781928] *** VALIDATE nfs4 *** [ 2.782861] pstore: using deflate compression [ 2.785769] Platform Keyring initialized [ 2.875543] NET: Registered protocol family 38 [ 2.876960] Key type asymmetric registered [ 2.878047] Asymmetric key parser 'x509' registered [ 2.879358] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.882043] io scheduler mq-deadline registered [ 2.885017] io scheduler kyber registered [ 2.886515] io scheduler bfq registered [ 2.888090] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.890269] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.892596] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.894598] ACPI: Power Button [PWRF] [ 2.899067] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.904976] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.944345] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.953533] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.995211] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.024021] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.051461] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.055438] Non-volatile memory driver v1.3 [ 3.056557] Linux agpgart interface v0.103 [ 3.085616] virtio_blk virtio1: [vda] 149976 512-byte logical blocks (76.8 MB/73.2 MiB) [ 3.087398] vda: detected capacity change from 0 to 76787712 [ 3.100910] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.102936] vdb: detected capacity change from 0 to 1073741824 [ 3.114981] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.117431] vdc: detected capacity change from 0 to 2621440000 [ 3.128402] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.130255] vdd: detected capacity change from 0 to 2621440000 [ 3.141944] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.144851] vde: detected capacity change from 0 to 4294967296 [ 3.159787] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.163840] vdf: detected capacity change from 0 to 4294967296 [ 3.170856] libphy: Fixed MDIO Bus: probed [ 3.177083] usbcore: registered new interface driver usbserial_generic [ 3.179436] usbserial: USB Serial support registered for generic [ 3.181828] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.185551] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.187371] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.189381] mousedev: PS/2 mouse device common for all mice [ 3.191334] rtc_cmos 00:05: RTC can wake from S4 [ 3.191416] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.194182] rtc_cmos 00:05: registered as rtc0 [ 3.197326] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.197610] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.201117] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.202359] intel_pstate: CPU model not supported [ 3.206538] hid: raw HID events driver (C) Jiri Kosina [ 3.207709] usbcore: registered new interface driver usbhid [ 3.208897] usbhid: USB HID core driver [ 3.209881] drop_monitor: Initializing network drop monitor service [ 3.211347] Initializing XFRM netlink socket [ 3.212622] NET: Registered protocol family 10 [ 3.214801] Segment Routing with IPv6 [ 3.215991] NET: Registered protocol family 17 [ 3.217473] mpls_gso: MPLS GSO support [ 3.221704] RAS: Correctable Errors collector initialized. [ 3.223368] AVX version of gcm_enc/dec engaged. [ 3.224315] AES CTR mode by8 optimization enabled [ 3.278026] sched_clock: Marking stable (3278010573, 0)->(4163216977, -885206404) [ 3.280368] registered taskstats version 1 [ 3.281984] Loading compiled-in X.509 certificates [ 3.283304] zswap: loaded using pool lzo/zbud [ 3.303641] Key type big_key registered [ 3.316607] Key type encrypted registered [ 3.317928] ima: No TPM chip found, activating TPM-bypass! [ 3.319376] ima: Allocated hash algorithm: sha1 [ 3.320396] ima: No architecture policies found [ 3.321570] evm: Initialising EVM extended attributes: [ 3.322654] evm: security.selinux [ 3.323285] evm: security.ima [ 3.323970] evm: security.capability [ 3.324779] evm: HMAC attrs: 0x1 [ 3.326361] rtc_cmos 00:05: setting system clock to 2026-09-14 08:46:17 UTC (1789375577) [ 3.331660] debug: unmapping init [mem 0xffffffff97403000-0xffffffff975fffff] [ 3.333627] debug: unmapping init [mem 0xffffffff96182000-0xffffffff96458fff] [ 3.344167] Write protecting the kernel read-only data: 28672k [ 3.346683] debug: unmapping init [mem 0xffffffff94803000-0xffffffff949fffff] [ 3.348526] debug: unmapping init [mem 0xffffffff95114000-0xffffffff951fffff] [ 3.381392] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.387144] systemd[1]: Detected virtualization kvm. [ 3.388379] systemd[1]: Detected architecture x86-64. [ 3.389587] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.411979] systemd[1]: No hostname configured. [ 3.413156] systemd[1]: Set hostname to . [ 3.414631] random: systemd: uninitialized urandom read (16 bytes read) [ 3.416121] systemd[1]: Initializing machine ID from random generator. [ 3.525993] random: systemd: uninitialized urandom read (16 bytes read) [ 3.528753] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.532452] random: systemd: uninitialized urandom read (16 bytes read) [ 3.536164] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.540588] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket (/dev/log). Starting Apply Kernel Variables... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Journal Service... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Slices. [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.084706] device-mapper: uevent: version 1.0.3 [ 4.087478] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.749352] virtio_net virtio0 ens2: renamed from eth0 [ 4.784773] random: fast init done [ 4.815083] scsi host0: ata_piix [ 4.836920] scsi host1: ata_piix [ 4.838431] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.841026] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.834559] random: crng init done [ 9.837202] random: 7 urandom warning(s) missed due to ratelimiting [ 10.505243] dracut-initqueue[577]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 12.691768] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 15.047343] printk: systemd: 26 output lines suppressed due to ratelimiting [ 15.890188] SELinux: Disabled at runtime. [ 15.991675] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 16.005864] systemd[1]: Detected virtualization kvm. [ 16.007541] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 17.266256] systemd[1]: initrd-switch-root.service: Succeeded. [ 17.271167] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 17.280282] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 17.289125] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 17.294934] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 17.302918] systemd[1]: Starting Journal Service... Starting Journal Service... [ 17.329307] systemd[1]: Listening on initctl Compatibility Named Pipe. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Create list of required st…ce nodes for the current kernel... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-getty.slice. Mounting Huge Pages File System... [ OK ] Listening on udev Control Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on RPCbind Server Activation Socket. Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Starting Remount Root and Kernel File Systems... [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on Process Core Dump Socket. Mounting POSIX Message Queue File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ 18.044661] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Kernel Debug File System... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 18.870044] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 19.852322] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 19.864895] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 20.152262] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 20.211238] EDAC sbridge: Ver: 1.1.2 [ 23.441781] Key type dns_resolver registered [ 24.133840] NFS: Registering the id_resolver key type [ 24.138147] Key type id_resolver registered [ 24.143366] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (7s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg247-server login: [ 73.737790] libcfs: loading out-of-tree module taints kernel. [ 73.775290] Key type ._llcrypt registered [ 73.777960] Key type .llcrypt registered [ 73.893508] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_hostid [ 90.677354] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [ 91.997507] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 92.008744] alg: No test for adler32 (adler32-zlib) [ 93.443162] Lustre: Lustre: Build Version: 2.17.58_40_gbbf22b4 [ 94.292886] LNet: Added LNI 192.168.202.147@tcp [8/256/0/180] [ 96.023377] Key type lgssc registered [ 97.663478] Lustre: Echo OBD driver; http://www.lustre.org/ [ 115.056816] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 153.112790] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [ 165.396912] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 165.445829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 166.704526] LustreError: 6507:0:(mgs_llog.c:673:mgs_find_or_make_fsdb_nolock()) lustre: New filesystem registration disabled. Use 'lctl set_param allow_register=unlimited' to allow new filesystem registrations: rc = -13 [ 166.713600] Lustre: First target registration detected: opening a one-time 24-hour registration window. Use 'lctl set_param allow_register=0' to close it early, or 'lctl set_param -P allow_register=0' to lock down permanently. [ 166.731216] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 166.750171] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 166.820841] Lustre: lustre-MDT0000: new disk, initializing [ 166.890249] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 166.919236] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 171.367536] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 176.615788] LustreError: 7038:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 178.383561] Lustre: New target registration enabled indefinitely [ 188.314953] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 188.408328] Lustre: 6504:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 188.434163] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 188.438257] Lustre: Skipped 1 previous similar message [ 188.509592] Lustre: lustre-MDT0001: new disk, initializing [ 188.560358] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 188.578942] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 188.583727] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 191.923569] hrtimer: interrupt took 2785551 ns [ 192.491504] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 196.912238] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 206.233813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 206.480564] Lustre: lustre-OST0000: new disk, initializing [ 206.488264] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 206.495132] Lustre: 8686:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 206.576119] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 212.906372] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 214.090864] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 214.115369] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 214.266341] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 226.553691] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 226.662861] Lustre: lustre-OST0001: new disk, initializing [ 226.666211] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 226.672259] Lustre: 9758:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 226.742640] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 231.975286] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 231.991744] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 232.118252] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 232.326629] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 246.161785] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 251.813388] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 257.391824] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing check_logdir /tmp/testlogs/ [ 261.992722] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing yml_node [ 264.948573] Lustre: DEBUG MARKER: Client: 2.17.58.40 [ 266.395849] Lustre: DEBUG MARKER: MDS: 2.17.58.40 [ 268.267232] Lustre: DEBUG MARKER: OSS: 2.17.58.40 [ 269.332469] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Sep 14 04:50:42 EDT 2026 [ 282.980853] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 285.407989] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 306.655957] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 306.670131] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 306.682056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 308.705733] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 308.707303] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 308.720229] Lustre: Skipped 1 previous similar message [ 308.739510] Lustre: Skipped 3 previous similar messages [ 310.759399] Lustre: server umount lustre-MDT0000 complete [ 313.915721] LustreError: 6498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789375888 with bad export cookie 6552259405724998762 [ 313.922898] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 313.928507] LustreError: 6498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 318.943612] LustreError: 6513:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 318.944164] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 318.955517] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 318.958894] LustreError: 6513:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 318.996482] Lustre: Skipped 2 previous similar messages [ 320.218081] Lustre: server umount lustre-MDT0001 complete [ 323.658460] Lustre: server umount lustre-OST0000 complete [ 327.113638] Lustre: server umount lustre-OST0001 complete [ 332.465813] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_hostid [ 339.677822] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [ 369.999335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 370.342515] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 370.408540] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 370.507501] Lustre: lustre-MDT0000: new disk, initializing [ 370.561112] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 370.578722] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 374.492359] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 385.096946] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 385.199698] Lustre: 15313:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 385.247077] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 385.251712] Lustre: Skipped 1 previous similar message [ 385.303582] Lustre: lustre-MDT0001: new disk, initializing [ 385.375193] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 385.421461] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 385.433231] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 389.104316] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 396.516126] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 401.382758] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 406.898907] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 407.162117] Lustre: lustre-OST0000: new disk, initializing [ 407.165908] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 407.172282] Lustre: 17328:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 407.242520] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 409.036309] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 409.047529] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 409.134338] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 412.010450] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 420.653245] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 424.617691] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 424.866607] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 428.412871] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 428.571505] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 434.656459] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 434.660597] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 434.669825] LustreError: Skipped 1 previous similar message [ 434.676258] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 434.685884] Lustre: Skipped 1 previous similar message [ 434.701058] Lustre: Skipped 2 previous similar messages [ 439.781808] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 439.790035] Lustre: Skipped 1 previous similar message [ 444.383381] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 444.503769] Lustre: server umount lustre-OST0000 complete [ 450.023098] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 450.026644] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 450.035775] Lustre: Skipped 1 previous similar message [ 450.048393] Lustre: Skipped 1 previous similar message [ 454.187802] Lustre: server umount lustre-MDT0000 complete [ 455.138256] LustreError: 15325:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 455.161334] LustreError: 15325:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 458.689471] LustreError: 15307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789376032 with bad export cookie 6552259405725000295 [ 458.689771] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 458.704792] LustreError: 15307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 459.233678] Lustre: server umount lustre-MDT0001 complete [ 463.895294] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 04:53:56 (1789376036) [ 466.462231] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 468.336900] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 04:54:01 (1789376041) [ 473.492485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 473.897962] LustreError: 19738:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 473.989953] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 477.369971] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 479.217375] LustreError: 19739:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 484.320487] LustreError: 19738:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 486.077332] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 490.639126] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 497.912833] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 502.786223] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 509.637036] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 516.620862] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 524.302832] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 530.750987] LustreError: 22314:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 535.013449] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 535.016773] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 535.020257] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 540.886787] Lustre: server umount lustre-OST0000 complete [ 550.252185] Lustre: server umount lustre-MDT0000 complete [ 550.372725] LustreError: 19734:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 550.394168] LustreError: 19734:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 553.388021] LustreError: 19718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789376127 with bad export cookie 6552259405725001177 [ 553.393990] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 553.396122] LustreError: 19718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 553.666558] Lustre: server umount lustre-MDT0001 complete [ 563.928627] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing unload_modules_local [ 568.738786] Key type lgssc unregistered [ 569.401612] LNet: 23464:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 569.415865] LNetError: 23464:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 569.455143] LNet: Removed LNI 192.168.202.147@tcp [ 571.262134] Key type .llcrypt unregistered [ 571.263419] Key type ._llcrypt unregistered [ 595.279142] Key type ._llcrypt registered [ 595.280440] Key type .llcrypt registered [ 595.351970] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [ 595.960531] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 596.029745] alg: No test for adler32 (adler32-zlib) [ 597.177491] Lustre: Lustre: Build Version: 2.17.58_40_gbbf22b4 [ 597.395350] LNet: Added LNI 192.168.202.147@tcp [8/256/0/180] [ 599.156477] Key type lgssc registered [ 600.241334] Lustre: Echo OBD driver; http://www.lustre.org/ [ 609.049419] loop: module loaded [ 609.114853] loop0: detected capacity change from 0 to 204800000 [ 609.365605] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 609.376889] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 609.744848] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 616.413797] loop0: detected capacity change from 0 to 204800000 [ 616.669051] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 616.681795] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 616.697174] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 622.390594] loop0: detected capacity change from 0 to 204800000 [ 628.449826] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 628.505757] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 629.906561] LustreError: 25627:0:(mgs_llog.c:673:mgs_find_or_make_fsdb_nolock()) lustre: New filesystem registration disabled. Use 'lctl set_param allow_register=unlimited' to allow new filesystem registrations: rc = -13 [ 629.920115] Lustre: First target registration detected: opening a one-time 24-hour registration window. Use 'lctl set_param allow_register=0' to close it early, or 'lctl set_param -P allow_register=0' to lock down permanently. [ 629.975471] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 630.078877] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 630.208109] Lustre: lustre-MDT0000: new disk, initializing [ 630.368081] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 630.402107] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 634.833451] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 644.662764] loop1: detected capacity change from 0 to 204800000 [ 651.996773] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 652.322796] Lustre: lustre-OST07c6: new disk, initializing [ 652.326225] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 652.331204] Lustre: Skipped 1 previous similar message [ 652.336452] Lustre: 26646:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 652.428042] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 658.574137] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 658.589457] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 658.628272] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 658.908631] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 664.278348] Lustre: 25661:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 664.285351] Lustre: 25661:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 664.290395] Lustre: 25661:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 664.295184] Lustre: 25661:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 664.300372] Lustre: 25661:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 664.303823] Lustre: 25661:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 678.004246] Lustre: 25633:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 678.013189] Lustre: 25633:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 678.020514] Lustre: 25633:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 678.028798] Lustre: 25633:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 678.044805] Lustre: 25633:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 678.054993] Lustre: 25633:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 682.963046] Lustre: 25634:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 682.980674] Lustre: 25634:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 682.987844] Lustre: 25634:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 682.999496] Lustre: 25634:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 683.014353] Lustre: 25634:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 683.019187] Lustre: 25634:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 683.024270] Lustre: 25634:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 683.030420] Lustre: 25634:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 683.034204] Lustre: 25634:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 683.040484] Lustre: 25634:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 683.046248] Lustre: 25634:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 683.050818] Lustre: 25634:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 686.248390] Lustre: 25661:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 686.254574] Lustre: 25661:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 686.264858] Lustre: 25661:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 686.277861] Lustre: 25661:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 686.285568] Lustre: 25661:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 686.294264] Lustre: 25661:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 686.302718] Lustre: 25661:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 686.310365] Lustre: 25661:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 686.316088] Lustre: 25661:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 686.321160] Lustre: 25661:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 686.326168] Lustre: 25661:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 686.335557] Lustre: 25661:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 691.176924] Lustre: 25633:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 691.188433] Lustre: 25633:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 691.198038] Lustre: 25633:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 691.204492] Lustre: 25633:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 691.212104] Lustre: 25633:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 691.222830] Lustre: 25633:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 691.233698] Lustre: 25633:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 691.247272] Lustre: 25633:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 691.254329] Lustre: 25633:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 691.260954] Lustre: 25633:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 691.267131] Lustre: 25633:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 691.275410] Lustre: 25633:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 702.276284] Lustre: 25634:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 702.285727] Lustre: 25634:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 702.291275] Lustre: 25634:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 702.295884] Lustre: 25634:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 702.304124] Lustre: 25634:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 702.315149] Lustre: 25634:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 702.323704] Lustre: 25634:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 702.332121] Lustre: 25634:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 702.338480] Lustre: 25634:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 702.343574] Lustre: 25634:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 702.348511] Lustre: 25634:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 702.353082] Lustre: 25634:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 755.698807] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 755.702721] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 755.750222] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 759.372291] Lustre: server umount lustre-OST07c6 complete [ 769.716758] Lustre: server umount lustre-MDT0000 complete [ 786.733028] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing unload_modules_local [ 789.560836] Key type lgssc unregistered [ 789.842872] LNet: 29201:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 789.848222] LNetError: 29201:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 789.873327] LNet: Removed LNI 192.168.202.147@tcp [ 790.624514] Key type .llcrypt unregistered [ 790.626322] Key type ._llcrypt unregistered [ 809.253460] Key type ._llcrypt registered [ 809.255420] Key type .llcrypt registered [ 809.377936] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [ 810.490417] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 810.631564] alg: No test for adler32 (adler32-zlib) [ 811.720863] Lustre: Lustre: Build Version: 2.17.58_40_gbbf22b4 [ 811.996480] LNet: Added LNI 192.168.202.147@tcp [8/256/0/180] [ 813.711374] Key type lgssc registered [ 814.507594] Lustre: Echo OBD driver; http://www.lustre.org/ [ 822.189617] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 822.225971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 823.759492] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 829.304948] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 833.021206] LustreError: 30479:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 833.044388] LustreError: 30479:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 834.381847] LustreError: 30949:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 836.738901] Lustre: New target registration enabled indefinitely [ 838.116210] LustreError: 30478:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 843.238665] LustreError: 30479:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 846.133153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 846.653986] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 850.114852] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 859.122294] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 866.137320] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 874.065264] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 874.373348] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 881.972811] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 890.842739] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 892.862266] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 910.949584] LustreError: 33472:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 917.484412] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 917.494835] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 917.504885] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 918.498613] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 918.504863] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 918.518853] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 921.908931] Lustre: server umount lustre-OST0000 complete [ 928.737849] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 928.738774] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 928.739270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 928.739275] Lustre: Skipped 1 previous similar message [ 928.753428] Lustre: Skipped 1 previous similar message [ 933.323788] Lustre: server umount lustre-MDT0000 complete [ 933.861719] LustreError: 30479:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 933.883727] LustreError: 30479:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 938.771904] LustreError: 30459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789376512 with bad export cookie 13615246761509726912 [ 938.772616] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 938.780745] LustreError: 30459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 939.982141] Lustre: server umount lustre-MDT0001 complete [ 954.519632] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing unload_modules_local [ 958.003530] Key type lgssc unregistered [ 958.230507] LNet: 34625:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 958.238907] LNetError: 34625:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 958.257417] LNet: Removed LNI 192.168.202.147@tcp [ 959.556517] Key type .llcrypt unregistered [ 959.565579] Key type ._llcrypt unregistered [ 985.187297] Key type ._llcrypt registered [ 985.189791] Key type .llcrypt registered [ 985.379943] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [ 986.740788] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 986.900994] alg: No test for adler32 (adler32-zlib) [ 988.181209] Lustre: Lustre: Build Version: 2.17.58_40_gbbf22b4 [ 988.529619] LNet: Added LNI 192.168.202.147@tcp [8/256/0/180] [ 990.311642] Key type lgssc registered [ 991.898342] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1002.618798] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1002.643773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1004.449977] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1009.771262] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 1014.612653] LustreError: 36366:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1014.765172] LustreError: 35894:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1014.790084] LustreError: 35894:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1017.043940] Lustre: New target registration enabled indefinitely [ 1019.876713] LustreError: 35895:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1024.998102] LustreError: 35894:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1027.046810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1027.379579] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1033.116972] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 1041.918921] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1049.328920] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1057.684997] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1058.063615] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1066.954239] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 1074.486225] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1077.122325] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1088.882400] LustreError: 38683:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1095.136738] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1095.153384] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1095.161804] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1095.174470] Lustre: Skipped 1 previous similar message [ 1095.189808] Lustre: Skipped 1 previous similar message [ 1100.318963] Lustre: server umount lustre-OST0000 complete [ 1105.380175] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1105.386590] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1105.407259] Lustre: Skipped 1 previous similar message [ 1105.426202] Lustre: Skipped 1 previous similar message [ 1110.496377] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1110.509022] Lustre: Skipped 1 previous similar message [ 1111.915467] Lustre: server umount lustre-MDT0000 complete [ 1115.619217] LustreError: 35890:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1115.645821] LustreError: 35890:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1117.612669] LustreError: 35876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789376691 with bad export cookie 1835158236814310553 [ 1117.627138] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1117.633221] LustreError: 35876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1118.226091] Lustre: server umount lustre-MDT0001 complete [ 1135.799167] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing unload_modules_local [ 1138.697447] Key type lgssc unregistered [ 1139.127937] LNet: 39854:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1139.135726] LNetError: 39854:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1139.164742] LNet: Removed LNI 192.168.202.147@tcp [ 1140.164486] Key type .llcrypt unregistered [ 1140.170475] Key type ._llcrypt unregistered [ 1168.759317] Key type ._llcrypt registered [ 1168.761654] Key type .llcrypt registered [ 1168.860368] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [ 1170.270103] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1170.451156] alg: No test for adler32 (adler32-zlib) [ 1171.599831] Lustre: Lustre: Build Version: 2.17.58_40_gbbf22b4 [ 1171.860608] LNet: Added LNI 192.168.202.147@tcp [8/256/0/180] [ 1173.551764] Key type lgssc registered [ 1175.150889] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1183.674340] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1183.687924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1185.211056] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1190.176870] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 1194.378407] LustreError: 41738:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1195.494374] LustreError: 41268:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1195.529226] LustreError: 41268:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1196.275221] Lustre: New target registration enabled indefinitely [ 1200.622989] LustreError: 41267:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1204.294994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1204.729588] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1209.761980] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 1219.596921] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1225.992304] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1233.978346] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1234.309138] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1240.824376] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 1250.065869] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1256.838225] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1266.949918] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.47@tcp (stopping) [ 1267.173892] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1267.178954] Lustre: Skipped 2 previous similar messages [ 1268.854222] Lustre: server umount lustre-MDT0000 complete [ 1272.066442] LustreError: 41262:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1274.162159] LustreError: 41249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789376848 with bad export cookie 6631508096507171550 [ 1274.169237] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1274.181446] LustreError: 41249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1277.170805] LustreError: 41262:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1277.196559] LustreError: 41262:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 1277.203377] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.47@tcp (stopping) [ 1277.203387] Lustre: Skipped 3 previous similar messages [ 1277.408649] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1279.458717] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1279.468439] Lustre: Skipped 1 previous similar message [ 1280.681634] Lustre: server umount lustre-MDT0001 complete [ 1304.779317] Lustre: server umount lustre-OST0000 complete [ 1320.865426] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing unload_modules_local [ 1324.166051] Key type lgssc unregistered [ 1324.646817] LNet: 45538:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1324.658644] LNetError: 45538:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1324.685763] LNet: Removed LNI 192.168.202.147@tcp [ 1325.890210] Key type .llcrypt unregistered [ 1325.895508] Key type ._llcrypt unregistered [ 1351.262347] Key type ._llcrypt registered [ 1351.264380] Key type .llcrypt registered [ 1351.345348] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [ 1352.370789] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1352.465145] alg: No test for adler32 (adler32-zlib) [ 1353.430214] Lustre: Lustre: Build Version: 2.17.58_40_gbbf22b4 [ 1353.618358] LNet: Added LNI 192.168.202.147@tcp [8/256/0/180] [ 1355.271480] Key type lgssc registered [ 1356.378509] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1364.753792] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1364.769169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1366.160746] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1371.487359] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 1376.737666] LustreError: 46952:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1376.781033] LustreError: 46952:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1377.056796] LustreError: 47421:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1378.969887] Lustre: New target registration enabled indefinitely [ 1381.862300] LustreError: 46951:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1386.185345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1386.481851] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1390.954382] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 1398.032703] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1404.379402] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1412.105075] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1412.528953] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1421.596436] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 1431.327314] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1436.020452] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1443.671541] Lustre: lustre-OST0000: Not available for connect from 192.168.202.47@tcp (stopping) [ 1446.380629] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1446.388988] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1446.411824] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1446.438897] Lustre: Skipped 1 previous similar message [ 1448.740496] Lustre: server umount lustre-OST0000 complete [ 1464.076807] LustreError: 49950:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1471.977341] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1472.001470] Lustre: Skipped 1 previous similar message [ 1472.004663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1472.034720] Lustre: Skipped 1 previous similar message [ 1477.092448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1477.106020] Lustre: Skipped 1 previous similar message [ 1477.825172] Lustre: server umount lustre-MDT0000 complete [ 1482.217357] LustreError: 46951:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1482.251931] LustreError: 46951:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1482.302349] LustreError: 46933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789377056 with bad export cookie 9259229219901639405 [ 1482.328473] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1482.336619] LustreError: 46933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1488.921628] Lustre: server umount lustre-MDT0001 complete [ 1501.303896] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing unload_modules_local [ 1504.608939] Key type lgssc unregistered [ 1504.942583] LNet: 51056:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1504.952890] LNetError: 51056:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1504.986087] LNet: Removed LNI 192.168.202.147@tcp [ 1506.041732] Key type .llcrypt unregistered [ 1506.044371] Key type ._llcrypt unregistered [ 1530.977698] Key type ._llcrypt registered [ 1530.980316] Key type .llcrypt registered [ 1531.117678] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [ 1532.586854] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1532.607236] alg: No test for adler32 (adler32-zlib) [ 1533.784069] Lustre: Lustre: Build Version: 2.17.58_40_gbbf22b4 [ 1534.129271] LNet: Added LNI 192.168.202.147@tcp [8/256/0/180] [ 1535.815229] Key type lgssc registered [ 1536.997316] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1543.906854] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1543.927857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1545.272163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1549.335311] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 1553.389867] LustreError: 52941:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1555.260494] Lustre: New target registration enabled indefinitely [ 1555.432104] LustreError: 52471:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1555.450286] LustreError: 52471:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1560.552395] LustreError: 52472:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1562.791869] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1563.129792] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1565.870319] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 1573.868106] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1580.698958] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1587.164849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1587.491881] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1590.573474] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1594.239901] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 1602.639210] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1608.364384] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1611.548457] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1623.556537] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-14 5h13m15s [ 1625.013410] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1629.668033] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1629.672884] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1629.687606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1630.756749] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1630.762471] Lustre: Skipped 5 previous similar messages [ 1633.467473] Lustre: server umount lustre-MDT0000 complete [ 1636.087958] LustreError: 52466:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1636.714318] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1642.226450] LustreError: 54145:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1642.246952] LustreError: 54145:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 1643.857446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1644.117609] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1644.513131] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1644.650913] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1645.552133] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1649.633944] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 1654.614571] Lustre: DEBUG MARKER: Wait for df (37301) ... [ 1656.052935] Lustre: DEBUG MARKER: done [ 1658.972859] LustreError: 56689:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1664.996888] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1665.011628] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1665.020766] Lustre: Skipped 2 previous similar messages [ 1665.025591] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1665.029919] Lustre: Skipped 1 previous similar message [ 1669.352929] Lustre: server umount lustre-OST0000 complete [ 1676.260767] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1676.265938] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1676.291100] LustreError: Skipped 1 previous similar message [ 1676.291803] Lustre: Skipped 1 previous similar message [ 1676.298891] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1676.298904] Lustre: Skipped 3 previous similar messages [ 1680.051560] Lustre: server umount lustre-MDT0000 complete [ 1683.428319] LustreError: 52472:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1683.465756] LustreError: 52472:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 1683.872070] LustreError: 53449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789377258 with bad export cookie 14389673272465562662 [ 1683.874211] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1683.892818] LustreError: 53449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1684.333921] Lustre: server umount lustre-MDT0001 complete [ 1696.451500] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing unload_modules_local [ 1698.575749] Key type lgssc unregistered [ 1698.861295] LNet: 57838:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1698.868387] LNetError: 57838:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1698.907681] LNet: Removed LNI 192.168.202.147@tcp [ 1699.742910] Key type .llcrypt unregistered [ 1699.745369] Key type ._llcrypt unregistered [ 1744.956152] Key type ._llcrypt registered [ 1744.957563] Key type .llcrypt registered [ 1745.173355] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [ 1746.701464] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1746.720936] alg: No test for adler32 (adler32-zlib) [ 1747.835075] Lustre: Lustre: Build Version: 2.17.58_40_gbbf22b4 [ 1748.139414] LNet: Added LNI 192.168.202.147@tcp [8/256/0/180] [ 1749.824102] Key type lgssc registered [ 1750.996436] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1761.672049] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1761.689119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1762.903378] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1762.910860] LustreError: 60051:0:(mgs_llog.c:673:mgs_find_or_make_fsdb_nolock()) lustre: New filesystem registration disabled. Use 'lctl set_param allow_register=unlimited' to allow new filesystem registrations: rc = -13 [ 1762.920758] LustreError: 60051:0:(mgs_handler.c:593:mgs_target_reg()) Can't get db for lustre: -13 [ 1762.927180] LustreError: lustre-MDT0000: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 1762.935942] LustreError: 60035:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [ 1762.940564] LustreError: 60035:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 1762.944618] LustreError: 60035:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 1762.993129] Lustre: server umount lustre-MDT0000 complete [ 1762.995850] LustreError: 60035:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 1764.638344] Lustre: DEBUG MARKER: conf-sanity test_35b: @@@@@@ FAIL: MDT start failed [ 1769.187861] Lustre: DEBUG MARKER: == conf-sanity test 36: df report consistency on OSTs with different block size ========================================================== 05:15:41 (1789377341) [ 1770.732861] Lustre: DEBUG MARKER: SKIP: conf-sanity test_36 remote OST [ 1772.279461] Lustre: DEBUG MARKER: == conf-sanity test 37: verify set tunables works for symlink device ========================================================== 05:15:45 (1789377345) [ 1785.135834] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [ 1790.990731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1791.373536] LustreError: 61077:0:(mgs_llog.c:673:mgs_find_or_make_fsdb_nolock()) lustre: New filesystem registration disabled. Use 'lctl set_param allow_register=unlimited' to allow new filesystem registrations: rc = -13 [ 1791.386948] Lustre: First target registration detected: opening a one-time 24-hour registration window. Use 'lctl set_param allow_register=0' to close it early, or 'lctl set_param -P allow_register=0' to lock down permanently. [ 1791.402319] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1791.416620] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1791.550968] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1792.540090] Lustre: Failing over lustre-MDT0000 [ 1792.787860] Lustre: server umount lustre-MDT0000 complete [ 1799.160735] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 05:16:11 (1789377371) [ 1805.325870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1806.102607] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1811.291058] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 1815.217760] LustreError: 62201:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1816.886685] Lustre: New target registration enabled indefinitely [ 1824.901117] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1825.092286] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1825.156855] Lustre: 61725:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1825.356856] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1829.787340] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 1836.483281] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1840.558569] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1847.698905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1847.821954] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1847.985458] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1854.474629] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 1857.713955] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1861.946806] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1868.784358] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1874.737398] Lustre: server umount lustre-MDT0000 complete [ 1876.454717] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1876.466892] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1878.502071] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1878.966590] LustreError: 61715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789377453 with bad export cookie 16952238363638122228 [ 1878.969749] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1878.976483] LustreError: 61715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1879.235867] Lustre: server umount lustre-MDT0001 complete [ 1882.439816] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1886.604607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1894.431511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1898.976951] Lustre: 59373:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789377457/real 1789377457] req@000000002f087a8b x1876298114832768/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789377473 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1899.002443] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1899.022480] Lustre: Skipped 1 previous similar message [ 1904.095164] Lustre: 59370:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789377462/real 1789377462] req@00000000ae842e4a x1876298114832896/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789377478 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1904.164729] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xeb4272f994664fde [ 1904.170881] Lustre: MGC192.168.202.147@tcp: Connection restored to 0@lo (at 0@lo) [ 1904.499535] LustreError: 63814:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1904.518141] LustreError: 63814:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 1904.581805] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1904.694450] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1905.569319] Lustre: 59371:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789377463/real 1789377463] req@000000007d6bed62 x1876298114833152/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789377479 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1908.727735] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 1909.669391] LustreError: 63815:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1909.683786] LustreError: 63815:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1911.799762] Lustre: 59372:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789377469/real 1789377469] req@00000000d6221233 x1876298114833280/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789377485 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1914.850073] LustreError: 63814:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1914.871324] LustreError: 63814:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1916.336993] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1916.749168] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1918.824122] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1920.891625] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 1928.098834] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1933.157942] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1942.497717] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1942.510289] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1942.543446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1944.547116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1944.553090] Lustre: Skipped 1 previous similar message [ 1946.827630] Lustre: server umount lustre-MDT0000 complete [ 1949.668890] LustreError: 65736:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1949.685314] LustreError: 65736:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1950.569910] LustreError: 61715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789377524 with bad export cookie 16952238363638124510 [ 1950.577352] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1950.589022] LustreError: 61715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1950.848872] Lustre: server umount lustre-MDT0001 complete [ 1957.242934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1967.518419] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1969.889825] Lustre: 59373:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789377528/real 1789377528] req@0000000029281793 x1876298114860288/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789377544 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1969.921455] Lustre: 59373:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1969.929882] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1969.943387] Lustre: Skipped 2 previous similar messages [ 1975.082485] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xeb4272f994665541 [ 1975.095949] Lustre: MGC192.168.202.147@tcp: Connection restored to 0@lo (at 0@lo) [ 1975.107785] Lustre: Skipped 1 previous similar message [ 1975.287425] LustreError: 63814:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1975.368178] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1975.436612] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1978.730574] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 1981.407175] Lustre: 59372:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789377539/real 1789377539] req@00000000554bf9ca x1876298114860800/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789377555 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1981.417757] Lustre: 59372:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 1985.506886] LustreError: 66641:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1985.528725] LustreError: 66641:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 1988.142875] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1993.542065] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 1993.730936] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1998.887122] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2000.917744] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2005.766752] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2013.299425] Lustre: server umount lustre-MDT0000 complete [ 2014.189800] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2014.190762] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2014.205483] LustreError: 66641:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2014.252020] Lustre: Skipped 2 previous similar messages [ 2016.993944] LustreError: 61715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789377591 with bad export cookie 16952238363638125889 [ 2017.000486] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2017.007162] LustreError: 61715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2017.527435] Lustre: server umount lustre-MDT0001 complete [ 2023.270171] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2026.779409] Lustre: DEBUG MARKER: files compared the same [ 2041.975817] Lustre: server umount lustre-OST0000 complete [ 2064.633510] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing unload_modules_local [ 2067.947688] Key type lgssc unregistered [ 2068.409202] LNet: 71937:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2068.416228] LNetError: 71937:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2068.430252] LNet: Removed LNI 192.168.202.147@tcp [ 2069.107722] Key type .llcrypt unregistered [ 2069.110350] Key type ._llcrypt unregistered [ 2098.507902] Key type ._llcrypt registered [ 2098.513897] Key type .llcrypt registered [ 2098.646350] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [ 2099.474462] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2099.491832] alg: No test for adler32 (adler32-zlib) [ 2100.443996] Lustre: Lustre: Build Version: 2.17.58_40_gbbf22b4 [ 2100.614663] LNet: Added LNI 192.168.202.147@tcp [8/256/0/180] [ 2102.231306] Key type lgssc registered [ 2103.992582] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2113.484788] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2113.506774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2114.897987] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2120.523467] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug +malloc all [ 2124.145343] LustreError: 73826:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 2125.280840] LustreError: 73356:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2125.295448] LustreError: 73356:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2125.641873] Lustre: New target registration enabled indefinitely [ 2130.405813] LustreError: 73355:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2133.496295] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2133.775249] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2137.212731] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug +malloc all [ 2147.945939] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2157.953609] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2166.703268] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2166.988946] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2171.892648] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug +malloc all [ 2180.787940] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2185.382080] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2193.417119] LustreError: 76150:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 2199.008733] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2199.022952] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2199.051814] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2199.685569] Lustre: server umount lustre-OST0000 complete [ 2205.663764] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2205.669966] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2205.690118] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2211.296842] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2211.306611] Lustre: Skipped 3 previous similar messages [ 2213.391405] Lustre: server umount lustre-MDT0000 complete [ 2216.416934] LustreError: 73356:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2216.425906] LustreError: 73356:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2217.163551] LustreError: 73337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789377791 with bad export cookie 10264771470334008788 [ 2217.164562] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2217.174976] LustreError: 73337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2217.492995] Lustre: server umount lustre-MDT0001 complete [ 2227.168818] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing unload_modules_local [ 2229.932182] Key type lgssc unregistered [ 2230.202290] LNet: 77301:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2230.206742] LNetError: 77301:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2231.275288] LNet: Removed LNI 192.168.202.147@tcp [ 2232.338502] Key type .llcrypt unregistered [ 2232.342320] Key type ._llcrypt unregistered [ 2257.600662] Key type ._llcrypt registered [ 2257.602955] Key type .llcrypt registered [ 2257.694438] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [ 2260.525265] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2260.571612] alg: No test for adler32 (adler32-zlib) [ 2261.970839] Lustre: Lustre: Build Version: 2.17.58_40_gbbf22b4 [ 2262.507272] LNet: Added LNI 192.168.202.147@tcp [8/256/0/180] [ 2264.359217] Key type lgssc registered [ 2265.509309] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2275.684507] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2275.790403] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2291.490477] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2296.607511] LustreError: 78669:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.147@tcp: failed processing log, type 4: rc = -110 [ 2323.295215] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2328.455256] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 2336.703951] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2345.903561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2346.125606] Lustre: *** cfs_fail_loc=706, val=0*** [ 2346.159602] LustreError: 78003:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000000b4dd2f6 x1876298653763840/t0(0) o250->MGC192.168.202.147@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2346.913734] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2347.094568] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2352.099280] LustreError: 78695:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2352.137484] LustreError: 78695:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2354.427894] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 2357.238053] LustreError: 78694:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2358.251346] LustreError: 78695:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2358.269579] LustreError: 78695:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2359.816957] LustreError: 80234:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 2362.180149] Lustre: New target registration enabled indefinitely [ 2363.374150] LustreError: 78694:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2363.398505] LustreError: 78694:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2368.492120] LustreError: 79755:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2369.236889] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2369.632989] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2373.529935] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 2382.485938] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2390.276961] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2391.996072] LustreError: 81525:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 2398.179237] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2398.190622] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2398.211819] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2400.228465] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2400.234504] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2400.264781] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2405.356766] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2405.369711] Lustre: Skipped 1 previous similar message [ 2410.469596] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2410.476106] Lustre: Skipped 1 previous similar message [ 2410.979156] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2411.204763] Lustre: server umount lustre-OST0000 complete [ 2415.586342] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2415.590140] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2415.624798] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2415.637611] Lustre: Skipped 1 previous similar message [ 2421.839891] Lustre: server umount lustre-MDT0000 complete [ 2425.824539] LustreError: 79756:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2425.834058] LustreError: 79756:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2425.844914] LustreError: 78677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789378000 with bad export cookie 1613912069367129230 [ 2425.846443] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2425.849000] LustreError: 78677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2426.404509] Lustre: server umount lustre-MDT0001 complete [ 2437.181818] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing unload_modules_local [ 2441.162762] Key type lgssc unregistered [ 2441.626635] LNet: 82681:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2441.633816] LNetError: 82681:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2441.672164] LNet: Removed LNI 192.168.202.147@tcp [ 2442.998484] Key type .llcrypt unregistered [ 2442.999988] Key type ._llcrypt unregistered [ 2470.169312] Key type ._llcrypt registered [ 2470.174444] Key type .llcrypt registered [ 2470.322410] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [ 2471.742209] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2471.761388] alg: No test for adler32 (adler32-zlib) [ 2472.869820] Lustre: Lustre: Build Version: 2.17.58_40_gbbf22b4 [ 2473.073194] LNet: Added LNI 192.168.202.147@tcp [8/256/0/180] [ 2474.705865] Key type lgssc registered [ 2476.720563] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2485.620664] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2485.658469] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2490.982120] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 2493.486523] LustreError: 84526:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 2494.850153] Lustre: New target registration enabled indefinitely [ 2500.349665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2500.748102] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2505.290580] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 2506.216251] LustreError: 85026:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2506.234335] LustreError: 85026:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2510.312310] LustreError: 85024:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2510.327034] LustreError: 85024:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2514.073983] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2514.425647] LustreError: 85026:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2514.457135] LustreError: 85026:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2514.497316] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2519.526188] LustreError: 85025:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2519.548715] LustreError: 85025:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2522.666847] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 2524.642730] LustreError: 85029:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2524.688252] LustreError: 85029:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2532.264796] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2532.306283] Lustre: MGS: Client 640b6acc-50aa-499e-916e-54c575615372 (at 0@lo) reconnecting [ 2532.327615] LustreError: 85040:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2532.537746] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2532.624901] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2536.840904] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 2553.319065] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2553.322668] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2553.338816] LustreError: Skipped 1 previous similar message [ 2553.343121] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2553.361507] Lustre: Skipped 1 previous similar message [ 2553.372433] Lustre: Skipped 1 previous similar message [ 2557.429687] Lustre: server umount lustre-OST0000 complete [ 2563.552373] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2563.557457] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2563.565728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2563.579782] Lustre: Skipped 1 previous similar message [ 2568.674048] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2568.676581] Lustre: Skipped 1 previous similar message [ 2573.792663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2573.796023] Lustre: Skipped 1 previous similar message [ 2576.351279] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2576.629440] Lustre: server umount lustre-MDT0000 complete [ 2578.912295] LustreError: 86995:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2578.944182] LustreError: 86995:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 2582.956413] Lustre: server umount lustre-MDT0001 complete [ 2589.666934] Lustre: server umount MGS complete [ 2602.474148] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 05:29:35 (1789378175) [ 2637.014727] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_hostid [ 2648.752713] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [ 2690.368421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2698.355640] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 2713.549113] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2713.702859] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2713.751110] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2713.816470] Lustre: lustre-MDT0001: new disk, initializing [ 2713.897249] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2713.934452] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2713.939972] Lustre: 92298:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2713.965834] LustreError: 92298:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2718.748650] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 2729.735864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2729.915703] Lustre: lustre-OST0000: new disk, initializing [ 2729.919905] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2729.925027] Lustre: 93067:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2729.990390] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2738.467563] Lustre: 93365:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000013554432 x1876298875053696/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1789378328 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2739.093294] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 2753.011760] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2759.136472] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2760.029550] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2760.047402] Lustre: MGS: Client 700f4862-48ca-45d0-b830-c84aef214574 (at 0@lo) reconnecting [ 2760.116773] Lustre: 91519:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2760.129290] LustreError: 92284:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2760.175842] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2760.234711] Lustre: lustre-MDT0000: new disk, initializing [ 2760.422088] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2760.445356] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2761.514630] Lustre: 94137:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000a33c0de7 x1876298875061120/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1789378351 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2765.238990] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 2766.630848] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2770.669417] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2770.688530] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2770.786074] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2794.467577] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2794.476255] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2798.621265] Lustre: server umount lustre-OST0000 complete [ 2804.706025] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2804.713869] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2804.727155] Lustre: Skipped 1 previous similar message [ 2804.731770] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2804.767346] Lustre: Skipped 2 previous similar messages [ 2809.563562] Lustre: server umount lustre-MDT0000 complete [ 2809.844850] LustreError: 92274:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2809.881967] LustreError: 92274:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2821.082790] Lustre: server umount lustre-MDT0001 complete [ 2825.664175] Lustre: server umount MGS complete [ 2841.041845] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 05:33:32 (1789378412) [ 2869.891425] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing unload_modules_local [ 2873.142717] Key type lgssc unregistered [ 2873.707233] LNet: 96831:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2873.731616] LNetError: 96831:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2874.798291] LNet: Removed LNI 192.168.202.147@tcp [ 2876.140566] Key type .llcrypt unregistered [ 2876.142831] Key type ._llcrypt unregistered [ 2902.431900] Key type ._llcrypt registered [ 2902.433266] Key type .llcrypt registered [ 2902.590121] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [ 2904.175876] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2904.211264] alg: No test for adler32 (adler32-zlib) [ 2905.228667] Lustre: Lustre: Build Version: 2.17.58_40_gbbf22b4 [ 2905.459947] LNet: Added LNI 192.168.202.147@tcp [8/256/0/180] [ 2907.135228] Key type lgssc registered [ 2908.132464] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2915.933253] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2915.955289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2917.732089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2922.133886] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 2926.605351] LustreError: 98520:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 2927.078740] LustreError: 98049:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2927.088122] LustreError: 98049:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2929.341529] Lustre: New target registration enabled indefinitely [ 2932.194616] LustreError: 98050:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2937.315458] LustreError: 98049:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2939.140160] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2939.520419] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2945.219533] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 2955.921875] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2964.509155] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2971.288723] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2971.700356] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2978.103831] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 2986.922209] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2987.066519] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2992.748474] LustreError: 100826:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 2997.216305] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2997.223641] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2997.228916] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2997.243334] Lustre: Skipped 1 previous similar message [ 2997.253855] Lustre: Skipped 1 previous similar message [ 3002.337634] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3002.351349] Lustre: Skipped 1 previous similar message [ 3002.795545] Lustre: server umount lustre-OST0000 complete [ 3007.460701] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3007.486405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3007.493054] Lustre: Skipped 1 previous similar message [ 3012.577170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3012.585453] Lustre: Skipped 1 previous similar message [ 3013.071441] Lustre: server umount lustre-MDT0000 complete [ 3017.148867] LustreError: 98031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789378591 with bad export cookie 1194272891025110978 [ 3017.170806] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3017.517546] Lustre: server umount lustre-MDT0001 complete [ 3028.046866] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing unload_modules_local [ 3030.891405] Key type lgssc unregistered [ 3031.236142] LNet: 101977:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3031.249288] LNetError: 101977:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3031.278337] LNet: Removed LNI 192.168.202.147@tcp [ 3032.092499] Key type .llcrypt unregistered [ 3032.094763] Key type ._llcrypt unregistered [ 3047.969464] Key type ._llcrypt registered [ 3047.974688] Key type .llcrypt registered [ 3048.077667] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [ 3049.628414] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3049.655801] alg: No test for adler32 (adler32-zlib) [ 3050.909180] Lustre: Lustre: Build Version: 2.17.58_40_gbbf22b4 [ 3051.144283] LNet: Added LNI 192.168.202.147@tcp [8/256/0/180] [ 3052.951281] Key type lgssc registered [ 3054.879735] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3062.786712] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3062.802935] LustreError: 103050:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 3062.949348] LustreError: 103060:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 3062.966884] LustreError: 103050:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4854 [ 3062.978262] LustreError: 103050:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3062.998796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3062.999246] LustreError: 103050:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3064.514386] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3072.296371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3072.686880] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3079.876288] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 3088.308580] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing lsmod [ 3091.551648] LustreError: 104267:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 3091.556344] LustreError: 104268:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 3091.567412] LustreError: 104267:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4990 [ 3091.578811] LustreError: 104267:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3091.587100] LustreError: 104267:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3091.593573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3092.245494] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3097.123406] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3097.134199] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3097.136019] Lustre: Skipped 1 previous similar message [ 3097.152712] Lustre: Skipped 2 previous similar messages [ 3101.332567] Lustre: server umount lustre-MDT0000 complete [ 3102.178549] LustreError: 103097:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3102.192708] LustreError: 103097:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 3104.172340] LustreError: 103535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789378678 with bad export cookie 714012377386087815 [ 3104.178444] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3104.182068] LustreError: 103535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3105.375311] LustreError: 103528:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000056521389 x1876299481109760/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3105.500345] Lustre: server umount lustre-MDT0001 complete [ 3118.933899] Lustre: server umount lustre-OST0000 complete [ 3131.540299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3132.415789] LustreError: 105330:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3132.503937] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3137.530095] LustreError: 105331:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3139.449609] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 3142.629288] LustreError: 105330:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3143.992970] LustreError: 105802:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 3145.812893] Lustre: New target registration enabled indefinitely [ 3148.772429] LustreError: 105331:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3148.801075] LustreError: 105331:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3153.871764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3154.193927] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3157.534163] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 3163.842969] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3170.596969] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3177.720340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3178.085208] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3179.160415] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3185.981504] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 3194.708249] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3202.033772] LustreError: 108107:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 3205.097167] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3205.102664] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3205.103225] LustreError: Skipped 1 previous similar message [ 3205.104149] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3205.123945] Lustre: Skipped 2 previous similar messages [ 3210.214153] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3210.216889] Lustre: Skipped 1 previous similar message [ 3211.409284] Lustre: server umount lustre-OST0000 complete [ 3215.342151] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3215.348203] LustreError: 102457:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000e91cb089 x1876299481141888/t0(0) o8->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3215.386627] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3215.393699] Lustre: Skipped 1 previous similar message [ 3220.448814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3220.457683] Lustre: Skipped 1 previous similar message [ 3221.964333] Lustre: server umount lustre-MDT0000 complete [ 3225.568984] LustreError: 105331:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3225.585406] LustreError: 105331:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3226.113748] LustreError: 106311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789378800 with bad export cookie 714012377386088508 [ 3226.129275] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3226.145305] LustreError: 106311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3226.566427] Lustre: server umount lustre-MDT0001 complete [ 3238.526425] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing unload_modules_local [ 3241.274618] Key type lgssc unregistered [ 3241.597195] LNet: 109257:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3241.604159] LNetError: 109257:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3241.640523] LNet: Removed LNI 192.168.202.147@tcp [ 3242.597801] Key type .llcrypt unregistered [ 3242.599339] Key type ._llcrypt unregistered [ 3266.676880] Key type ._llcrypt registered [ 3266.678782] Key type .llcrypt registered [ 3266.766725] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [ 3267.858302] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3267.885621] alg: No test for adler32 (adler32-zlib) [ 3269.056288] Lustre: Lustre: Build Version: 2.17.58_40_gbbf22b4 [ 3269.367524] LNet: Added LNI 192.168.202.147@tcp [8/256/0/180] [ 3271.063268] Key type lgssc registered [ 3272.431804] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3281.273881] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3281.301743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3282.801722] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3287.946123] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 3292.138753] LustreError: 110668:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3292.198702] LustreError: 110668:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3293.149081] LustreError: 111138:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 3295.520841] Lustre: New target registration enabled indefinitely [ 3297.260340] LustreError: 110667:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3302.375318] LustreError: 110668:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3305.036185] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3305.540268] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3312.275609] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 3323.070203] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3330.352048] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3337.125991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3337.559079] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3344.286757] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 3351.930374] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3354.940445] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3361.464295] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3367.994232] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3371.931533] LustreError: 113551:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 3377.632905] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3377.644448] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3377.647318] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3377.657534] Lustre: Skipped 1 previous similar message [ 3382.771239] LustreError: 113357:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3382.993710] Lustre: server umount lustre-OST0000 complete [ 3387.361655] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3387.367488] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3387.377146] Lustre: Skipped 1 previous similar message [ 3387.381203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3392.994506] LustreError: 111742:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3393.021941] LustreError: 111742:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3393.170935] Lustre: server umount lustre-MDT0000 complete [ 3397.192106] LustreError: 110647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789378971 with bad export cookie 7867129405127338762 [ 3397.196116] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3397.206145] LustreError: 110647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3397.667845] Lustre: server umount lustre-MDT0001 complete [ 3409.614642] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing unload_modules_local [ 3412.041307] Key type lgssc unregistered [ 3412.490836] LNet: 114701:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3412.501408] LNetError: 114701:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3412.523976] LNet: Removed LNI 192.168.202.147@tcp [ 3413.451368] Key type .llcrypt unregistered [ 3413.453164] Key type ._llcrypt unregistered [ 3438.774477] Key type ._llcrypt registered [ 3438.779386] Key type .llcrypt registered [ 3438.968820] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [ 3441.039105] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3441.498308] alg: No test for adler32 (adler32-zlib) [ 3443.022957] Lustre: Lustre: Build Version: 2.17.58_40_gbbf22b4 [ 3443.944247] LNet: Added LNI 192.168.202.147@tcp [8/256/0/180] [ 3445.919338] Key type lgssc registered [ 3448.109113] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3459.443290] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3459.474053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3461.044125] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3466.076699] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 3470.743300] LustreError: 116396:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 3471.331828] LustreError: 115925:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3471.373487] LustreError: 115925:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3472.370524] LustreError: 115926:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3472.956485] Lustre: New target registration enabled indefinitely [ 3477.484639] LustreError: 115925:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3482.596809] LustreError: 115926:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3483.061671] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3483.433491] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3488.593954] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 3497.588697] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3504.400737] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3511.531933] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3511.904090] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3518.597329] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 3526.011644] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3532.415214] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3543.643508] LustreError: 118719:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 3548.129202] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3548.140625] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3548.145633] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3548.150965] Lustre: Skipped 1 previous similar message [ 3549.242858] Lustre: server umount lustre-OST0000 complete [ 3553.248564] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3553.258459] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3553.268837] Lustre: Skipped 2 previous similar messages [ 3558.368287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3558.383593] Lustre: Skipped 1 previous similar message [ 3559.645324] Lustre: server umount lustre-MDT0000 complete [ 3563.488662] LustreError: 115922:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3563.505587] LustreError: 115922:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3563.639824] LustreError: 115906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789379137 with bad export cookie 11121845580380914505 [ 3563.663200] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3563.938583] Lustre: server umount lustre-MDT0001 complete [ 3575.915537] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing unload_modules_local [ 3578.819478] Key type lgssc unregistered [ 3579.099280] LNet: 119869:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3579.127697] LNetError: 119869:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3580.198605] LNet: Removed LNI 192.168.202.147@tcp [ 3580.882944] Key type .llcrypt unregistered [ 3580.885095] Key type ._llcrypt unregistered [ 3607.036605] Key type ._llcrypt registered [ 3607.038791] Key type .llcrypt registered [ 3607.176067] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [ 3608.426489] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3608.524421] alg: No test for adler32 (adler32-zlib) [ 3609.942000] Lustre: Lustre: Build Version: 2.17.58_40_gbbf22b4 [ 3610.315444] LNet: Added LNI 192.168.202.147@tcp [8/256/0/180] [ 3612.087308] Key type lgssc registered [ 3613.683947] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3621.710811] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3621.732775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3623.019238] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3626.936644] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 3630.906770] LustreError: 121752:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 3632.661537] Lustre: New target registration enabled indefinitely [ 3633.635240] LustreError: 121282:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3633.646489] LustreError: 121282:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3638.759029] LustreError: 121281:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3640.025260] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3640.386261] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3644.862324] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 3652.625624] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3659.052654] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3665.553048] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3665.836192] LustreError: 123346:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3665.974890] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3668.033076] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3671.972654] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 3679.463294] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3686.244283] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3689.969253] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3689.973804] Lustre: Skipped 2 previous similar messages [ 3695.260237] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3695.291130] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3695.295650] Lustre: Skipped 2 previous similar messages [ 3695.304975] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3700.827110] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3708.416791] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.47@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3708.424445] Lustre: Skipped 2 previous similar messages [ 3710.686434] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.47@tcp 192.168.0.[2,10]@tcp [ 3716.527172] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.47/24@tcp in log lustre-MDT0000 [ 3716.535855] Lustre: Skipped 2 previous similar messages [ 3723.867556] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.47/24@tcp [ 3723.877233] Lustre: Skipped 1 previous similar message [ 3727.286750] LustreError: 125421:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 3734.499789] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3734.505019] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3734.506773] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3734.511540] Lustre: Skipped 1 previous similar message [ 3734.528050] LustreError: Skipped 1 previous similar message [ 3737.937991] Lustre: server umount lustre-OST0000 complete [ 3742.690790] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3742.700608] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3742.718446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3742.725573] Lustre: Skipped 1 previous similar message [ 3744.742508] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3744.750473] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3748.470685] Lustre: server umount lustre-MDT0000 complete [ 3749.859197] LustreError: 124620:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3749.890960] LustreError: 124620:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3751.817460] LustreError: 124154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789379325 with bad export cookie 2388014284611698100 [ 3751.822731] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3751.826963] LustreError: 124154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3752.093757] Lustre: server umount lustre-MDT0001 complete [ 3761.835518] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing unload_modules_local [ 3764.289274] Key type lgssc unregistered [ 3764.636981] LNet: 126571:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3764.644064] LNetError: 126571:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3764.658089] LNet: Removed LNI 192.168.202.147@tcp [ 3765.647592] Key type .llcrypt unregistered [ 3765.650969] Key type ._llcrypt unregistered [ 3802.614048] Key type ._llcrypt registered [ 3802.617996] Key type .llcrypt registered [ 3802.691373] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [ 3804.769578] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3804.860855] alg: No test for adler32 (adler32-zlib) [ 3806.069692] Lustre: Lustre: Build Version: 2.17.58_40_gbbf22b4 [ 3806.338922] LNet: Added LNI 192.168.202.147@tcp [8/256/0/180] [ 3808.047186] Key type lgssc registered [ 3809.125329] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3816.279902] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3816.301249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3817.782572] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3817.807665] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.47/24@tcp [ 3817.845179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3822.623449] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 3826.057492] LustreError: 128603:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 3827.170850] LustreError: 128130:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3827.195868] LustreError: 128130:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3827.951811] Lustre: New target registration enabled indefinitely [ 3832.289623] LustreError: 128131:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3836.129633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3836.408767] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3836.448891] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.47/24@tcp [ 3836.487642] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3839.784708] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 3846.406534] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3853.316038] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3861.322451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3861.666936] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3862.753120] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3868.216764] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 3876.399826] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3887.269966] LustreError: 130960:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 3891.683129] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3891.684719] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3891.706667] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3891.728819] Lustre: Skipped 1 previous similar message [ 3896.815780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3896.818324] Lustre: Skipped 1 previous similar message [ 3897.849300] Lustre: server umount lustre-OST0000 complete [ 3901.933330] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3901.945086] Lustre: Skipped 2 previous similar messages [ 3901.946942] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3901.953464] Lustre: Skipped 1 previous similar message [ 3907.040628] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3907.049378] Lustre: Skipped 1 previous similar message [ 3912.166340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3912.175647] Lustre: Skipped 1 previous similar message [ 3916.255192] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3916.419939] Lustre: server umount lustre-MDT0000 complete [ 3917.288182] LustreError: 128130:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3917.318330] LustreError: 128130:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3920.116955] LustreError: 128110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789379494 with bad export cookie 9798047904702550810 [ 3920.146081] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3920.476595] Lustre: server umount lustre-MDT0001 complete [ 3934.125399] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing unload_modules_local [ 3937.211491] Key type lgssc unregistered [ 3937.492727] LNet: 132113:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3937.502817] LNetError: 132113:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3938.536908] LNet: Removed LNI 192.168.202.147@tcp [ 3939.506127] Key type .llcrypt unregistered [ 3939.514607] Key type ._llcrypt unregistered [ 3967.283300] Key type ._llcrypt registered [ 3967.289430] Key type .llcrypt registered [ 3967.378284] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_hostid [ 3982.540935] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [ 3984.222728] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3984.500052] alg: No test for adler32 (adler32-zlib) [ 3985.822696] Lustre: Lustre: Build Version: 2.17.58_40_gbbf22b4 [ 3986.283352] LNet: Added LNI 192.168.202.147@tcp [8/256/0/180] [ 3988.047233] Key type lgssc registered [ 3989.643182] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4027.029714] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4027.040153] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4028.436252] LustreError: 135514:0:(mgs_llog.c:673:mgs_find_or_make_fsdb_nolock()) lustre: New filesystem registration disabled. Use 'lctl set_param allow_register=unlimited' to allow new filesystem registrations: rc = -13 [ 4028.448188] Lustre: First target registration detected: opening a one-time 24-hour registration window. Use 'lctl set_param allow_register=0' to close it early, or 'lctl set_param -P allow_register=0' to lock down permanently. [ 4028.462944] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4028.496727] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4028.666041] Lustre: lustre-MDT0000: new disk, initializing [ 4028.855088] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4028.918914] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4033.517310] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 4041.258293] LustreError: 136047:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 4043.701200] Lustre: New target registration enabled indefinitely [ 4051.710918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4051.946068] Lustre: 135514:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4052.098737] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4052.106021] Lustre: Skipped 1 previous similar message [ 4052.277210] Lustre: lustre-MDT0001: new disk, initializing [ 4052.511890] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4052.610540] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4052.629277] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4057.634584] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 4066.372665] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4072.013451] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4080.145813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4080.662568] Lustre: lustre-OST0000: new disk, initializing [ 4080.673034] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4080.685049] Lustre: 137725:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4080.786983] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4088.434801] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 4089.984207] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4089.990949] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4090.086043] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4099.594345] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4103.860653] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4104.016308] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4108.277579] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4108.651609] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4110.820492] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4110.830293] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4110.847053] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4110.850501] Lustre: Skipped 1 previous similar message [ 4115.943148] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4115.953373] Lustre: Skipped 1 previous similar message [ 4121.060510] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4121.070863] Lustre: Skipped 1 previous similar message [ 4125.152750] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4125.656566] Lustre: server umount lustre-OST0000 complete [ 4134.372305] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4134.392360] LustreError: Skipped 1 previous similar message [ 4134.400099] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4134.418987] Lustre: Skipped 1 previous similar message [ 4134.426842] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4136.420186] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4139.078163] Lustre: server umount lustre-MDT0000 complete [ 4143.136919] LustreError: 135504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789379717 with bad export cookie 120857208208426185 [ 4143.140295] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4143.147183] LustreError: 135504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4143.422775] Lustre: server umount lustre-MDT0001 complete [ 4152.291356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4153.149645] LustreError: 139943:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4153.165787] LustreError: 139943:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4153.222859] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4158.438975] LustreError: 139944:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4159.211378] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 4163.553808] LustreError: 139943:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4168.693979] LustreError: 139944:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4170.114713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4170.601852] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4176.638553] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 4186.304569] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4191.695505] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4202.041964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4202.870689] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4212.856805] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 4223.917311] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4232.161137] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4232.435072] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4239.803216] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4240.149778] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4250.247732] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4250.448579] Lustre: lustre-OST0001: new disk, initializing [ 4250.460477] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4250.468462] Lustre: 143365:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4250.618805] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4258.424450] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 4258.493437] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4258.521073] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4258.621420] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4268.541786] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4268.877589] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4274.514677] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4274.688443] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4281.711725] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff949b47267800.ost_server_uuid 50 [ 4283.657725] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff949b47267800.ost_server_uuid in IDLE state after 0 sec [ 4295.681925] Lustre: server umount lustre-OST0001 complete [ 4299.747109] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4299.758671] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4299.806634] LustreError: 141818:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4299.851255] LustreError: 141818:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4302.304541] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4302.311959] LustreError: Skipped 1 previous similar message [ 4302.318178] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4302.321340] Lustre: Skipped 2 previous similar messages [ 4304.876170] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4304.889695] Lustre: Skipped 2 previous similar messages [ 4306.898570] Lustre: server umount lustre-OST0000 complete [ 4314.080510] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4314.102609] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4314.131921] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4314.147577] Lustre: Skipped 2 previous similar messages [ 4317.764441] Lustre: server umount lustre-MDT0000 complete [ 4320.224703] LustreError: 139938:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4320.252177] LustreError: 139938:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4323.337792] LustreError: 139924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789379897 with bad export cookie 120857208208427060 [ 4323.346296] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4323.349310] LustreError: 139924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4323.609343] Lustre: server umount lustre-MDT0001 complete [ 4355.576087] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing unload_modules_local [ 4360.155872] Key type lgssc unregistered [ 4360.533562] LNet: 146538:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4360.540703] LNetError: 146538:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4360.554698] LNet: Removed LNI 192.168.202.147@tcp [ 4361.742778] Key type .llcrypt unregistered [ 4361.746765] Key type ._llcrypt unregistered [ 4412.407171] Key type ._llcrypt registered [ 4412.409971] Key type .llcrypt registered [ 4412.512253] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_hostid [ 4429.528705] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [ 4431.829661] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4431.876762] alg: No test for adler32 (adler32-zlib) [ 4433.148350] Lustre: Lustre: Build Version: 2.17.58_40_gbbf22b4 [ 4433.464449] LNet: Added LNI 192.168.202.147@tcp [8/256/0/180] [ 4435.167312] Key type lgssc registered [ 4436.767610] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4479.596071] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4479.630919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4481.127643] LustreError: 150694:0:(mgs_llog.c:673:mgs_find_or_make_fsdb_nolock()) lustre: New filesystem registration disabled. Use 'lctl set_param allow_register=unlimited' to allow new filesystem registrations: rc = -13 [ 4481.144507] Lustre: First target registration detected: opening a one-time 24-hour registration window. Use 'lctl set_param allow_register=0' to close it early, or 'lctl set_param -P allow_register=0' to lock down permanently. [ 4481.195834] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4481.287140] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4481.500326] Lustre: lustre-MDT0000: new disk, initializing [ 4481.687454] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4481.731431] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4488.897828] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 4496.691687] LustreError: 151241:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 4499.213258] Lustre: New target registration enabled indefinitely [ 4509.801107] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4509.987727] Lustre: 150694:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4510.067282] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4510.073715] Lustre: Skipped 1 previous similar message [ 4510.232189] Lustre: lustre-MDT0001: new disk, initializing [ 4510.378430] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4510.458336] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4510.473062] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4516.725250] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 4528.151314] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4535.575641] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4543.937238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4544.170256] Lustre: lustre-OST0000: new disk, initializing [ 4544.173467] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4544.177692] Lustre: 152917:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4544.253499] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4551.922549] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 4553.360050] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4553.368314] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4553.471750] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4562.067551] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4567.074764] Lustre: Failing over lustre-OST0000 [ 4567.678150] Lustre: server umount lustre-OST0000 complete [ 4585.447447] Lustre: 148918:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789380143/real 1789380143] req@00000000fa879b1b x1876300930256000/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789380159 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4585.448278] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4585.506650] Lustre: 148918:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4589.589415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4590.039350] Lustre: 148918:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789380148/real 1789380148] req@000000009275f8f0 x1876300930257408/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789380164 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4590.079232] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4590.091537] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4591.141684] Lustre: 148916:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789380148/real 1789380148] req@000000008cd2954f x1876300930257280/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789380164 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4595.173814] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4595.522542] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4595.523813] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4596.575567] Lustre: 148918:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789380153/real 1789380153] req@00000000647bd208 x1876300930258560/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789380169 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4597.999265] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 4603.215121] Lustre: Failing over lustre-MDT0000 [ 4603.589898] Lustre: server umount lustre-MDT0000 complete [ 4605.410785] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4605.434986] Lustre: Skipped 3 previous similar messages [ 4610.533551] LustreError: 150707:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4610.575509] LustreError: 150707:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 4615.654207] LustreError: 150701:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4615.698771] LustreError: 150701:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4620.771602] LustreError: 150706:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4620.798543] LustreError: 150706:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4621.858130] Lustre: 148917:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789380179/real 1789380179] req@00000000bd1906e1 x1876300930270208/t0(0) o400->MGC192.168.202.147@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789380195 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4621.913140] Lustre: 148917:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 4621.946367] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4627.443704] LustreError: 150707:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4627.504760] LustreError: 150707:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 4629.067647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4631.796577] LustreError: 150702:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4631.830704] LustreError: 150702:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4632.550324] LustreError: 148915:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000003de01bf8 x1876300930276480/t0(0) o250->MGC192.168.202.147@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4633.214850] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4633.318479] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4636.917445] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4638.713610] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4638.723434] Lustre: Skipped 1 previous similar message [ 4638.754893] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 4638.829393] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4640.367590] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 4652.589909] LustreError: 155570:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 4659.169966] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4659.175750] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4659.193908] Lustre: Skipped 1 previous similar message [ 4659.200313] Lustre: Skipped 1 previous similar message [ 4664.256645] Lustre: server umount lustre-OST0000 complete [ 4669.410575] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4669.417700] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4669.428827] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4669.454194] Lustre: Skipped 1 previous similar message [ 4674.532707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4674.548716] Lustre: Skipped 2 previous similar messages [ 4675.784534] Lustre: server umount lustre-MDT0000 complete [ 4679.649672] LustreError: 150707:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4679.669312] LustreError: 150707:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 4680.555301] LustreError: 155533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789380254 with bad export cookie 2181122583532924149 [ 4680.559325] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4680.564684] LustreError: 155533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4680.880888] Lustre: server umount lustre-MDT0001 complete [ 4694.131293] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing unload_modules_local [ 4697.492966] Key type lgssc unregistered [ 4697.971733] LNet: 156721:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4697.986513] LNetError: 156721:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4698.027927] LNet: Removed LNI 192.168.202.147@tcp [ 4699.171576] Key type .llcrypt unregistered [ 4699.176223] Key type ._llcrypt unregistered [ 4752.185641] Key type ._llcrypt registered [ 4752.193926] Key type .llcrypt registered [ 4752.466768] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [ 4755.740981] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4755.891274] alg: No test for adler32 (adler32-zlib) [ 4757.147993] Lustre: Lustre: Build Version: 2.17.58_40_gbbf22b4 [ 4757.411247] LNet: Added LNI 192.168.202.147@tcp [8/256/0/180] [ 4759.104792] Key type lgssc registered [ 4760.083665] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4769.242549] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4769.262493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4770.774376] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4775.100224] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 4780.004492] LustreError: 158241:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4780.022344] LustreError: 158241:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4785.125602] LustreError: 158242:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4786.296357] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4790.243563] LustreError: 158241:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4790.689595] LustreError: 158270:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4790.712890] LustreError: 158270:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4790.736019] LustreError: 158270:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 20, retries 0, failed: rc = -5 [ 4791.654611] Lustre: server umount lustre-MDT0000 complete [ 4809.985653] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_hostid [ 4825.587076] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [ 4863.141138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4863.405472] LustreError: 161597:0:(mgs_llog.c:673:mgs_find_or_make_fsdb_nolock()) lustre: New filesystem registration disabled. Use 'lctl set_param allow_register=unlimited' to allow new filesystem registrations: rc = -13 [ 4863.432490] Lustre: First target registration detected: opening a one-time 24-hour registration window. Use 'lctl set_param allow_register=0' to close it early, or 'lctl set_param -P allow_register=0' to lock down permanently. [ 4863.477263] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4863.569597] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4863.672361] Lustre: lustre-MDT0000: new disk, initializing [ 4863.801935] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4863.829927] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4868.568703] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 4875.200613] LustreError: 162129:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 4877.695806] Lustre: New target registration enabled indefinitely [ 4885.460941] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4885.626068] Lustre: 161595:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4885.653417] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4885.655984] Lustre: Skipped 1 previous similar message [ 4885.728279] Lustre: lustre-MDT0001: new disk, initializing [ 4885.841539] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4885.901453] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4885.906541] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4891.174850] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 4901.783290] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4908.224851] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4916.363917] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4916.737391] Lustre: lustre-OST0000: new disk, initializing [ 4916.740560] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4916.748405] Lustre: 163807:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4916.850593] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4924.072474] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 4927.081622] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4927.093224] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4927.229133] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4934.590986] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4939.942425] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4942.204464] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 06:08:34 (1789380514) [ 5193.185160] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5193.219629] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5193.360936] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5197.421518] Lustre: server umount lustre-MDT0000 complete [ 5198.822492] LustreError: 163208:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5198.875149] LustreError: 163208:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5202.092072] LustreError: 161588:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789380776 with bad export cookie 12376896072997530099 [ 5202.111190] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5202.111687] LustreError: 161588:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5203.940070] LustreError: 163814:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5203.940923] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5203.962963] Lustre: Skipped 2 previous similar messages [ 5203.972070] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5203.980042] Lustre: Skipped 2 previous similar messages [ 5207.968759] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5208.460569] Lustre: server umount lustre-MDT0001 complete [ 5218.664743] Lustre: server umount lustre-OST0000 complete [ 5232.042860] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_hostid [ 5246.346968] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [ 5286.120738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5286.325249] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5286.349730] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5286.420706] Lustre: lustre-MDT0000: new disk, initializing [ 5286.492519] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5286.547827] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5290.979742] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 5307.260554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5307.548773] Lustre: 168470:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5307.667543] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5307.675916] Lustre: Skipped 1 previous similar message [ 5307.815591] Lustre: lustre-MDT0001: new disk, initializing [ 5307.978981] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5308.065488] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5308.080642] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5315.732813] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 5327.582199] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5332.400150] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5338.361388] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5338.514045] Lustre: lustre-OST0000: new disk, initializing [ 5338.517260] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5338.522436] Lustre: 170489:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5338.568916] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5345.965635] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5345.980747] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5346.129573] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5346.595745] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 5360.412608] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5366.446852] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5366.856735] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5372.654903] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5372.970902] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5376.994328] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5376.994384] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5377.019786] LustreError: Skipped 1 previous similar message [ 5377.023076] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5377.092356] Lustre: Skipped 1 previous similar message [ 5381.940492] Lustre: server umount lustre-OST0000 complete [ 5389.792932] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5389.806597] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5389.826150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5389.828152] Lustre: Skipped 3 previous similar messages [ 5393.256470] Lustre: server umount lustre-MDT0000 complete [ 5395.424780] LustreError: 168482:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5395.440125] LustreError: 168482:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5396.743152] LustreError: 168461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789380970 with bad export cookie 12376896072997688922 [ 5396.754087] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5396.766028] LustreError: 168461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5396.998047] Lustre: server umount lustre-MDT0001 complete [ 5406.583806] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 06:16:19 (1789380979) [ 5431.311374] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_hostid [ 5440.620572] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [ 5481.347146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5481.738704] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5481.797167] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5481.911743] Lustre: lustre-MDT0000: new disk, initializing [ 5482.028607] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5482.055737] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5487.424268] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 5501.998992] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5502.215286] Lustre: 175248:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5502.697642] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5508.094549] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 5517.558616] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5523.257390] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5532.089987] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5532.367819] Lustre: lustre-OST0000: new disk, initializing [ 5532.369652] Lustre: Skipped 1 previous similar message [ 5532.373218] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5532.375835] Lustre: Skipped 2 previous similar messages [ 5532.379813] Lustre: 177276:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5540.475297] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5540.487764] Lustre: Skipped 1 previous similar message [ 5540.501810] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5540.626244] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5542.474812] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 5552.270777] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5566.437311] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5566.438245] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5566.454343] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5566.484431] Lustre: Skipped 2 previous similar messages [ 5566.494534] Lustre: Skipped 2 previous similar messages [ 5568.961525] Lustre: server umount lustre-OST0000 complete [ 5574.116641] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5576.672862] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5576.703028] Lustre: Skipped 1 previous similar message [ 5579.894612] Lustre: server umount lustre-MDT0000 complete [ 5581.801375] LustreError: 175255:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5581.823595] LustreError: 175255:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5584.366702] LustreError: 175239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789381158 with bad export cookie 12376896072997689811 [ 5584.369665] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5584.394544] LustreError: 175239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5584.991739] Lustre: server umount lustre-MDT0001 complete [ 5597.840144] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 06:19:29 (1789381169) [ 5616.835827] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_hostid [ 5627.604337] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [ 5669.900873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5670.108608] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5670.124817] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5670.171935] Lustre: lustre-MDT0000: new disk, initializing [ 5670.220831] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5670.224205] Lustre: Skipped 2 previous similar messages [ 5670.231679] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5675.779623] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 5689.411799] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5689.602841] Lustre: 181601:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5690.158976] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5696.183928] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 5707.273447] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5715.468511] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5725.023300] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5725.601850] Lustre: 183628:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5731.943068] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5732.167950] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5734.069154] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 5745.234427] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5753.520611] LustreError: 184481:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 5757.924061] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5757.926422] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5757.938184] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5757.938193] Lustre: Skipped 4 previous similar messages [ 5757.972945] Lustre: Skipped 1 previous similar message [ 5764.149495] Lustre: server umount lustre-OST0000 complete [ 5775.600054] Lustre: server umount lustre-MDT0000 complete [ 5778.401975] LustreError: 181612:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5778.423340] LustreError: 181612:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5779.641312] LustreError: 181593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789381353 with bad export cookie 12376896072997690896 [ 5779.645824] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5779.649659] LustreError: 181593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5780.252428] Lustre: server umount lustre-MDT0001 complete [ 5794.180745] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing unload_modules_local [ 5797.670440] Key type lgssc unregistered [ 5797.954861] LNet: 185631:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5797.963736] LNetError: 185631:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5799.016041] LNet: Removed LNI 192.168.202.147@tcp [ 5799.781124] Key type .llcrypt unregistered [ 5799.783830] Key type ._llcrypt unregistered [ 5826.999623] Key type ._llcrypt registered [ 5827.003197] Key type .llcrypt registered [ 5827.153500] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [ 5828.840795] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5828.883116] alg: No test for adler32 (adler32-zlib) [ 5830.048339] Lustre: Lustre: Build Version: 2.17.58_40_gbbf22b4 [ 5830.416425] LNet: Added LNI 192.168.202.147@tcp [8/256/0/180] [ 5832.143746] Key type lgssc registered [ 5833.762560] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5842.658303] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5842.714434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5844.394892] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5849.758580] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 5853.895975] LustreError: 187518:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 5854.703185] LustreError: 187048:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5854.731242] LustreError: 187048:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5855.916269] Lustre: New target registration enabled indefinitely [ 5859.818211] LustreError: 187049:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5863.954393] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5864.373224] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5869.645201] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 5878.259302] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5884.968543] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5892.031602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5892.561791] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5899.510476] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 5911.241469] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5914.052936] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5934.515817] LustreError: 189845:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 5939.688128] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5939.689782] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5939.718183] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5939.722686] Lustre: Skipped 1 previous similar message [ 5944.811491] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5944.823262] Lustre: Skipped 1 previous similar message [ 5945.754614] Lustre: server umount lustre-OST0000 complete [ 5951.459025] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5951.486120] LustreError: Skipped 1 previous similar message [ 5951.506919] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5951.528619] Lustre: Skipped 1 previous similar message [ 5951.542961] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5955.042698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5955.050499] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5958.451750] Lustre: server umount lustre-MDT0000 complete [ 5960.184527] LustreError: 187044:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5960.225965] LustreError: 187044:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5963.952066] LustreError: 187030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789381538 with bad export cookie 1428246792047244418 [ 5963.957293] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5963.981150] LustreError: 187030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5964.418605] Lustre: server umount lustre-MDT0001 complete [ 5979.040389] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing unload_modules_local [ 5982.208063] Key type lgssc unregistered [ 5982.502171] LNet: 190996:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5982.519843] LNetError: 190996:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5982.549201] LNet: Removed LNI 192.168.202.147@tcp [ 5983.815670] Key type .llcrypt unregistered [ 5983.822973] Key type ._llcrypt unregistered [ 6011.756548] Key type ._llcrypt registered [ 6011.758714] Key type .llcrypt registered [ 6011.868823] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [ 6013.146248] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6013.331588] alg: No test for adler32 (adler32-zlib) [ 6014.464555] Lustre: Lustre: Build Version: 2.17.58_40_gbbf22b4 [ 6014.674441] LNet: Added LNI 192.168.202.147@tcp [8/256/0/180] [ 6016.352847] Key type lgssc registered [ 6017.472095] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6025.778897] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6025.804186] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6027.159904] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6032.251836] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 6036.356499] LustreError: 192883:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 6037.485344] LustreError: 192412:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6037.531030] LustreError: 192412:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6038.379384] Lustre: New target registration enabled indefinitely [ 6042.597170] LustreError: 192413:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6046.043752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6046.488472] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6051.289128] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 6059.475590] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6066.064802] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6072.484507] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6072.759206] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6073.776326] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6079.503328] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 6089.696260] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6096.364447] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6096.375940] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6096.377554] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6096.389745] Lustre: Skipped 1 previous similar message [ 6097.673226] Lustre: lustre-OST0000: Not available for connect from 192.168.202.47@tcp (stopping) [ 6097.681848] Lustre: Skipped 1 previous similar message [ 6101.475110] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6102.446908] Lustre: server umount lustre-OST0000 complete [ 6112.720593] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff949b470de800.ost_server_uuid 50 [ 6114.853913] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6131.185036] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6131.199320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6131.204464] Lustre: Skipped 1 previous similar message [ 6135.839864] Lustre: server umount lustre-MDT0000 complete [ 6136.292855] LustreError: 192412:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6136.301293] LustreError: 192412:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6139.921330] LustreError: 192394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789381714 with bad export cookie 11721353721650671875 [ 6139.932310] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6139.936598] LustreError: 192394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6140.936956] Lustre: server umount lustre-MDT0001 complete [ 6152.873088] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 06:28:45 (1789381725) [ 6160.833305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6161.322324] LustreError: 196573:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6161.480267] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6166.499419] LustreError: 196574:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6166.663650] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 6176.743382] LustreError: 196573:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6176.758719] LustreError: 196573:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6177.711908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6178.119073] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6183.370676] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 6192.379669] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6199.509160] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6207.649152] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6208.152918] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6215.963736] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 6224.374795] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6231.822929] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6231.828724] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6232.135996] Lustre: lustre-OST0001: new disk, initializing [ 6232.141390] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6232.148607] Lustre: 199464:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6232.326269] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6239.468129] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 6241.908437] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6241.916890] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6241.977453] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6248.688582] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6252.516332] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6252.524253] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6252.531290] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6252.541582] Lustre: Skipped 2 previous similar messages [ 6252.552829] Lustre: Skipped 1 previous similar message [ 6257.946680] Lustre: server umount lustre-OST0000 complete [ 6259.954682] LustreError: 198445:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6265.696725] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6265.911483] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6271.683647] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6272.007829] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6288.353761] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6288.357181] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6288.366239] LustreError: Skipped 1 previous similar message [ 6288.368548] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6288.368556] Lustre: Skipped 3 previous similar messages [ 6288.418217] Lustre: Skipped 1 previous similar message [ 6292.969896] Lustre: server umount lustre-OST0001 complete [ 6298.594216] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6308.835137] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6308.842591] Lustre: Skipped 6 previous similar messages [ 6311.391224] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6311.924884] Lustre: server umount lustre-MDT0000 complete [ 6313.956631] LustreError: 196574:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6313.987198] LustreError: 196574:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 17 previous similar messages [ 6316.184229] LustreError: 196555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789381890 with bad export cookie 11721353721650672953 [ 6316.189755] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6316.200260] LustreError: 196555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6316.784625] Lustre: server umount lustre-MDT0001 complete [ 6341.529749] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 06:31:54 (1789381914) [ 6349.065977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6349.361625] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6349.404609] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6349.788962] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6355.373277] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 6366.225907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6366.317325] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6366.359576] Lustre: 202909:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6371.577864] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 6379.872277] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6385.771426] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6392.941510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6393.191902] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6400.126809] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 6400.642072] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6408.957396] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6416.715086] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6416.795141] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6417.064877] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6417.075490] Lustre: Skipped 2 previous similar messages [ 6423.475129] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 6431.591648] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6436.321657] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6436.327559] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6436.337667] Lustre: Skipped 1 previous similar message [ 6436.341771] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6440.455938] Lustre: server umount lustre-OST0000 complete [ 6442.741368] LustreError: 204809:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6457.824389] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6457.828570] LustreError: Skipped 1 previous similar message [ 6457.840643] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6457.864782] Lustre: Skipped 1 previous similar message [ 6463.925977] Lustre: server umount lustre-OST0001 complete [ 6474.690078] Lustre: server umount lustre-MDT0000 complete [ 6478.681494] LustreError: 202901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789382052 with bad export cookie 11721353721650674360 [ 6478.709469] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6478.711431] LustreError: 202901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6479.032839] Lustre: server umount lustre-MDT0001 complete [ 6507.551068] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 06:34:40 (1789382080) [ 6526.601718] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_hostid [ 6539.994319] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [ 6582.446293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6582.831511] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6582.878501] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6582.965556] Lustre: lustre-MDT0000: new disk, initializing [ 6583.106436] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6583.148606] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6588.813625] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 6601.356445] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6601.470414] Lustre: 211059:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6601.609911] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6601.616519] Lustre: Skipped 1 previous similar message [ 6601.763655] Lustre: lustre-MDT0001: new disk, initializing [ 6602.018576] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6602.036100] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6607.530918] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 6618.389164] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6624.551899] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6632.176676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6632.541659] Lustre: lustre-OST0000: new disk, initializing [ 6632.546602] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6632.555284] Lustre: 213077:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6640.302563] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 6641.840852] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6641.860410] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6641.970021] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6650.797947] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6656.131599] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6656.415405] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6661.306210] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6661.638424] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6667.756421] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6667.763674] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6667.764098] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6667.764103] LustreError: Skipped 1 previous similar message [ 6667.776847] Lustre: Skipped 4 previous similar messages [ 6667.807041] Lustre: Skipped 14 previous similar messages [ 6670.081611] Lustre: server umount lustre-OST0000 complete [ 6679.855389] Lustre: server umount lustre-MDT0000 complete [ 6683.118253] LustreError: 211070:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6683.144506] LustreError: 211070:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 13 previous similar messages [ 6684.034052] LustreError: 211051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789382258 with bad export cookie 11721353721650675914 [ 6684.044515] LustreError: 211051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6684.045357] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6684.500171] Lustre: server umount lustre-MDT0001 complete [ 6694.327375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6699.162862] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 6708.948522] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6713.455842] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 6721.159408] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6726.655382] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6733.615121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6740.566372] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 6748.228956] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6753.186568] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6753.397418] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6758.942930] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6759.139922] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6760.930020] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6760.940790] Lustre: Skipped 2 previous similar messages [ 6760.948831] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6760.955372] LustreError: Skipped 1 previous similar message [ 6775.263730] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6775.559116] Lustre: server umount lustre-OST0000 complete [ 6783.445391] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6788.665558] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6819.901458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6826.679228] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6826.711329] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6826.727783] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6826.743650] Lustre: Skipped 1 previous similar message [ 6827.886902] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 6837.068615] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6839.052512] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6847.462993] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6847.477949] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6847.485042] Lustre: Skipped 10 previous similar messages [ 6849.203176] Lustre: server umount lustre-OST0000 complete [ 6859.568818] Lustre: server umount lustre-MDT0000 complete [ 6864.391201] LustreError: 215272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789382438 with bad export cookie 11721353721650676803 [ 6864.414501] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6864.416792] LustreError: 215272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6864.879191] Lustre: server umount lustre-MDT0001 complete [ 6874.624923] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 06:40:47 (1789382447) [ 6881.600920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6882.094698] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6882.103455] Lustre: Skipped 6 previous similar messages [ 6886.858036] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 6896.760180] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6903.206291] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 6911.381593] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6917.780302] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6924.888768] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6932.184861] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 6940.259054] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6945.666792] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6945.931180] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6951.061194] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6951.189409] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6958.591260] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6958.972657] Lustre: lustre-OST0001: new disk, initializing [ 6958.975611] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6958.982966] Lustre: 224727:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6966.445167] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6966.451909] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6966.571805] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6967.260348] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 6978.364784] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6984.157561] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6984.407603] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6990.628977] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6991.023531] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6995.455108] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6995.474619] Lustre: Skipped 5 previous similar messages [ 7007.199420] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7007.486853] Lustre: server umount lustre-OST0001 complete [ 7010.789865] LustreError: 223183:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7010.815562] LustreError: 223183:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [ 7013.766884] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7013.968689] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7018.871648] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7019.192979] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7049.511153] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7054.832732] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7054.861393] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7054.861926] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7054.895974] Lustre: Skipped 1 previous similar message [ 7057.138442] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 7065.129374] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7066.815240] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7070.182112] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7070.192819] LustreError: Skipped 1 previous similar message [ 7075.360937] Lustre: server umount lustre-OST0001 complete [ 7099.152250] Lustre: server umount lustre-OST0000 complete [ 7106.019112] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7106.029192] Lustre: Skipped 12 previous similar messages [ 7109.863815] Lustre: server umount lustre-MDT0000 complete [ 7114.174966] LustreError: 222094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789382688 with bad export cookie 11721353721650677769 [ 7114.195939] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7114.200149] LustreError: 222094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7121.054475] Lustre: server umount lustre-MDT0001 complete [ 7148.734500] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 06:45:21 (1789382721) [ 7156.318557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7156.627652] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7156.660311] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7163.399371] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 7173.561578] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7173.718880] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7173.779476] Lustre: 229980:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7179.698727] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 7187.505312] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7194.753859] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7202.349220] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7202.644569] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7209.386201] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 7217.342095] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7227.915869] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7228.035264] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7237.241944] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 7246.024697] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7247.682378] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7252.882673] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7253.117326] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7257.256220] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7257.433417] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7263.273435] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff949b47262800.ost_server_uuid 50 [ 7264.749329] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff949b47262800.ost_server_uuid in IDLE state after 0 sec [ 7266.078440] Lustre: Permanently deactivating lustre-OST0001 [ 7266.124216] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7268.705629] Lustre: Permanently reactivating lustre-OST0001 [ 7274.983384] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7274.984932] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7275.018684] Lustre: Skipped 8 previous similar messages [ 7275.048970] LustreError: Skipped 1 previous similar message [ 7280.770486] Lustre: server umount lustre-OST0001 complete [ 7291.826742] Lustre: server umount lustre-OST0000 complete [ 7310.303155] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7310.413824] Lustre: server umount lustre-MDT0000 complete [ 7313.482189] LustreError: 229973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789382887 with bad export cookie 11721353721650679057 [ 7313.494918] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7313.499754] LustreError: 229973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7313.835797] Lustre: server umount lustre-MDT0001 complete [ 7340.496259] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 06:48:33 (1789382913) [ 7348.537461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7348.915089] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7348.938528] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7348.948627] Lustre: Skipped 5 previous similar messages [ 7353.991498] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 7364.032729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7364.172607] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7364.246351] Lustre: 236726:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7369.436248] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 7378.681454] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7385.336236] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7393.497298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7393.748595] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7393.778638] Lustre: Permanently deactivating lustre-OST0000 [ 7393.798401] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7401.173334] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 7401.534882] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7409.927851] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7417.970433] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7418.083845] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7418.378402] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7418.386736] Lustre: Skipped 11 previous similar messages [ 7425.136691] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 7427.267980] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7432.784150] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7434.446602] Lustre: Permanently reactivating lustre-OST0000 [ 7434.459258] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7434.464181] Lustre: Skipped 2 previous similar messages [ 7436.712460] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 35 seconds. I think it's dead, and I am evicting it. exp 00000000e9f6cf05, cur 1789383010 deadline 1789383005 last 1789382975 [ 7436.712491] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7436.730862] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7436.752483] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7436.822649] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7459.876153] Lustre: server umount lustre-OST0001 complete [ 7463.557488] LustreError: 240656:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 7475.438931] Lustre: server umount lustre-OST0000 complete [ 7486.451116] Lustre: server umount lustre-MDT0000 complete [ 7490.068169] LustreError: 237515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789383064 with bad export cookie 11721353721650680737 [ 7490.075018] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7490.089904] LustreError: 237515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7490.522608] Lustre: server umount lustre-MDT0001 complete [ 7504.175410] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing unload_modules_local [ 7507.610838] Key type lgssc unregistered [ 7507.951197] LNet: 241805:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7507.967171] LNetError: 241805:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7507.988808] LNet: Removed LNI 192.168.202.147@tcp [ 7509.245713] Key type .llcrypt unregistered [ 7509.248572] Key type ._llcrypt unregistered [ 7536.445243] Key type ._llcrypt registered [ 7536.450739] Key type .llcrypt registered [ 7536.570926] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [ 7538.360321] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7538.433560] alg: No test for adler32 (adler32-zlib) [ 7539.560145] Lustre: Lustre: Build Version: 2.17.58_40_gbbf22b4 [ 7539.942816] LNet: Added LNI 192.168.202.147@tcp [8/256/0/180] [ 7541.665359] Key type lgssc registered [ 7543.160710] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7554.506780] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7554.516354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7556.153378] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7561.541508] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 7565.892526] LustreError: 243743:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 7566.305296] LustreError: 243269:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7566.323097] LustreError: 243269:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7567.697302] Lustre: New target registration enabled indefinitely [ 7571.424935] LustreError: 243270:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7574.717130] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7574.846371] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7574.878749] Lustre: 243258:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7574.888815] Lustre: Permanently deactivating lustre-MDT0001 [ 7574.908658] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7575.217708] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7580.335084] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 7583.789340] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7590.589587] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7598.926262] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7607.056648] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7607.716870] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7614.906108] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 7620.020725] LustreError: 245357:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7620.045905] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7623.702951] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7625.191583] LustreError: 245357:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7626.264477] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7631.026053] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7631.461781] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7636.991288] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7636.998348] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7638.259605] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 7646.502786] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7650.485435] Lustre: Permanently reactivating lustre-MDT0001 [ 7650.498189] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7650.501977] Lustre: Skipped 2 previous similar messages [ 7657.237701] Lustre: lustre-MDT0001: Client 28072f92-0c07-4d2e-b38c-fa9292d0346f (at 192.168.202.47@tcp) reconnecting [ 7660.165873] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7660.179098] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7660.186734] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7660.198285] LustreError: 247172:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 7660.198548] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7660.206564] LustreError: 247172:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 7660.206694] LustreError: 247172:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.147@tcp: cfg command failed: rc = -108 [ 7660.206716] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 7660.206716] [ 7660.248936] LustreError: 243261:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 7663.399338] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7665.149654] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7670.811831] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7671.021768] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7677.791417] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7678.266160] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7681.892880] Lustre: Permanently deactivating lustre-MDT0001 [ 7681.906795] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7681.915730] Lustre: Skipped 2 previous similar messages [ 7689.889038] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7698.403373] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7698.413452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7698.432256] Lustre: Skipped 1 previous similar message [ 7703.528433] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7703.534839] Lustre: Skipped 3 previous similar messages [ 7704.884240] Lustre: server umount lustre-MDT0000 complete [ 7708.658950] LustreError: 243270:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7708.710891] LustreError: 243270:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 7710.232320] LustreError: 247065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789383284 with bad export cookie 1111563937287530760 [ 7710.251156] LustreError: 247065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7710.260892] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7711.250679] Lustre: server umount lustre-MDT0001 complete [ 7716.544192] Lustre: server umount lustre-OST0000 complete [ 7730.111063] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 06:55:02 (1789383302) [ 7742.947088] Lustre: server umount lustre-OST0001 complete [ 7751.286705] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_hostid [ 7763.485572] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [ 7804.016997] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7804.338565] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7804.357467] Lustre: Skipped 2 previous similar messages [ 7804.386483] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7804.498112] Lustre: lustre-MDT0000: new disk, initializing [ 7804.618671] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7804.644089] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7809.931175] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 7824.020892] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7824.165121] Lustre: 251778:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7824.241818] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7824.246649] Lustre: Skipped 1 previous similar message [ 7824.356425] Lustre: lustre-MDT0001: new disk, initializing [ 7824.495056] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7824.561972] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7824.591973] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7830.243966] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 7842.757776] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7849.264673] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7857.722325] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7858.088456] Lustre: lustre-OST0000: new disk, initializing [ 7858.095473] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7858.106303] Lustre: 253798:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7858.272214] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7863.886372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7863.915581] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7864.058232] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7865.794848] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 7876.487295] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7878.958310] LustreError: 251786:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7888.978189] LustreError: 251786:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7889.003610] LustreError: 254678:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7890.652255] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7890.843787] Lustre: lustre-OST0001: new disk, initializing [ 7890.847802] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7890.855340] Lustre: 254912:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7890.969737] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7898.650809] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 7898.746216] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7898.764063] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7898.874922] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7899.047155] LustreError: 254678:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7910.578679] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7912.661848] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7919.591298] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7919.597465] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7919.620225] LustreError: Skipped 1 previous similar message [ 7919.641340] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7919.650048] Lustre: Skipped 3 previous similar messages [ 7919.668657] Lustre: Skipped 4 previous similar messages [ 7921.118708] Lustre: server umount lustre-OST0001 complete [ 7924.705869] LustreError: 253781:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7924.744470] LustreError: 253781:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7938.405512] LustreError: 255942:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 7945.207321] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7945.208504] LustreError: 253784:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7945.215497] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7945.230645] Lustre: Skipped 1 previous similar message [ 7945.244883] LustreError: 253784:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 7948.342927] Lustre: server umount lustre-OST0000 complete [ 7955.429199] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7955.430907] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7955.454299] Lustre: Skipped 1 previous similar message [ 7955.468330] Lustre: Skipped 1 previous similar message [ 7959.245994] Lustre: server umount lustre-MDT0000 complete [ 7963.717139] LustreError: 251770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789383537 with bad export cookie 1111563937287533574 [ 7963.735837] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7963.738468] LustreError: 251770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7970.364812] Lustre: server umount lustre-MDT0001 complete [ 7982.970751] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing unload_modules_local [ 7985.977543] Key type lgssc unregistered [ 7986.292529] LNet: 257093:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7986.312592] LNetError: 257093:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7986.344228] LNet: Removed LNI 192.168.202.147@tcp [ 7987.247876] Key type .llcrypt unregistered [ 7987.249570] Key type ._llcrypt unregistered [ 8035.965242] Key type ._llcrypt registered [ 8035.966497] Key type .llcrypt registered [ 8036.026737] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [ 8036.751114] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8036.837231] alg: No test for adler32 (adler32-zlib) [ 8038.174946] Lustre: Lustre: Build Version: 2.17.58_40_gbbf22b4 [ 8038.541192] LNet: Added LNI 192.168.202.147@tcp [8/256/0/180] [ 8040.305455] Key type lgssc registered [ 8041.712556] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8050.467067] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8050.495327] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8051.896491] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8051.902629] LustreError: 259297:0:(mgs_llog.c:673:mgs_find_or_make_fsdb_nolock()) lustre: New filesystem registration disabled. Use 'lctl set_param allow_register=unlimited' to allow new filesystem registrations: rc = -13 [ 8051.912429] LustreError: 259297:0:(mgs_handler.c:593:mgs_target_reg()) Can't get db for lustre: -13 [ 8051.919263] LustreError: lustre-MDT0000: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 8051.924839] LustreError: 259280:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [ 8051.928852] LustreError: 259280:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 8051.931832] LustreError: 259280:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 8051.992521] Lustre: server umount lustre-MDT0000 complete [ 8051.995546] LustreError: 259280:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 8053.879915] Lustre: DEBUG MARKER: conf-sanity test_52: @@@@@@ FAIL: Unable to start MDS [ 8059.962325] Lustre: DEBUG MARKER: == conf-sanity test 53a: check OSS thread count params === 07:00:32 (1789383632) [ 8067.227491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8067.469657] LustreError: 259927:0:(mgs_llog.c:673:mgs_find_or_make_fsdb_nolock()) lustre: New filesystem registration disabled. Use 'lctl set_param allow_register=unlimited' to allow new filesystem registrations: rc = -13 [ 8067.478586] Lustre: First target registration detected: opening a one-time 24-hour registration window. Use 'lctl set_param allow_register=0' to close it early, or 'lctl set_param -P allow_register=0' to lock down permanently. [ 8067.506572] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8067.529506] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8067.681960] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8073.293072] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 8077.801030] LustreError: 260411:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 8079.980795] Lustre: New target registration enabled indefinitely [ 8088.555363] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8088.712655] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8088.768578] Lustre: 259927:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8089.232596] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8095.069215] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 8103.625630] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8109.195700] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8117.229131] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8117.412594] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8117.605340] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8124.463985] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 8127.112572] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8133.379072] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8158.679793] LustreError: 263529:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 8162.787103] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8162.797549] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8162.812742] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8163.305134] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8163.311866] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8163.345939] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8163.356723] Lustre: Skipped 1 previous similar message [ 8165.037287] Lustre: server umount lustre-OST0000 complete [ 8170.979506] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8170.983111] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8170.997150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8173.537514] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8173.545913] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8173.586843] Lustre: Skipped 1 previous similar message [ 8175.546142] Lustre: server umount lustre-MDT0000 complete [ 8179.797109] LustreError: 259920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789383753 with bad export cookie 11187858780950514120 [ 8179.807179] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8179.814735] LustreError: 259920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8180.476071] Lustre: server umount lustre-MDT0001 complete [ 8191.601223] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing unload_modules_local [ 8194.130360] Key type lgssc unregistered [ 8194.515172] LNet: 264678:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8194.519839] LNetError: 264678:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8194.533851] LNet: Removed LNI 192.168.202.147@tcp [ 8195.580525] Key type .llcrypt unregistered [ 8195.587390] Key type ._llcrypt unregistered [ 8217.384978] Key type ._llcrypt registered [ 8217.396407] Key type .llcrypt registered [ 8217.565681] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [ 8219.279339] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8219.349201] alg: No test for adler32 (adler32-zlib) [ 8220.497117] Lustre: Lustre: Build Version: 2.17.58_40_gbbf22b4 [ 8220.778615] LNet: Added LNI 192.168.202.147@tcp [8/256/0/180] [ 8222.536071] Key type lgssc registered [ 8224.277786] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8235.286600] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8235.322509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8237.059722] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8243.016356] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 8247.265243] LustreError: 265896:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8247.282514] LustreError: 265896:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8247.671236] LustreError: 266366:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 8250.222750] Lustre: New target registration enabled indefinitely [ 8252.393548] LustreError: 265895:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8257.533918] LustreError: 265896:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8259.823398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8260.215720] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8266.106816] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 8275.368510] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8283.103233] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8291.896501] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8292.222505] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8300.868184] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 8307.601813] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 8310.203876] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8321.967912] LustreError: 268813:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 8326.119552] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8326.131416] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8326.156594] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8328.161766] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8328.179071] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8328.188794] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8328.213496] Lustre: Skipped 1 previous similar message [ 8332.032985] Lustre: server umount lustre-OST0000 complete [ 8336.865246] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8336.879571] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8336.902781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8342.645490] Lustre: server umount lustre-MDT0000 complete [ 8343.525740] LustreError: 265896:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8343.550739] LustreError: 265896:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8346.423975] LustreError: 269012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789383920 with bad export cookie 12422791591819914874 [ 8346.424663] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8346.438927] LustreError: 269012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8346.890874] Lustre: server umount lustre-MDT0001 complete [ 8362.003434] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing unload_modules_local [ 8365.763298] Key type lgssc unregistered [ 8366.305508] LNet: 269967:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8366.317820] LNetError: 269967:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8366.344865] LNet: Removed LNI 192.168.202.147@tcp [ 8367.413373] Key type .llcrypt unregistered [ 8367.415260] Key type ._llcrypt unregistered [ 8396.771134] Key type ._llcrypt registered [ 8396.774196] Key type .llcrypt registered [ 8397.036424] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [ 8398.476456] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8398.548811] alg: No test for adler32 (adler32-zlib) [ 8399.753906] Lustre: Lustre: Build Version: 2.17.58_40_gbbf22b4 [ 8400.049070] LNet: Added LNI 192.168.202.147@tcp [8/256/0/180] [ 8401.839361] Key type lgssc registered [ 8403.331994] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8411.656697] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8411.705701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8413.268693] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8419.011300] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 8423.416986] LustreError: 271380:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8423.452499] LustreError: 271380:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8423.896507] LustreError: 271852:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 8425.985128] Lustre: New target registration enabled indefinitely [ 8428.533595] LustreError: 271381:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8433.635959] LustreError: 271380:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8436.003493] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8436.450041] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8442.000450] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 8450.829503] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8458.869779] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8467.511259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8467.895110] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8475.274722] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 8483.256671] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8483.261152] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8508.073606] LustreError: 274928:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 8514.029460] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8514.032765] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8514.048599] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8514.070360] Lustre: Skipped 1 previous similar message [ 8514.101450] Lustre: Skipped 1 previous similar message [ 8514.450247] Lustre: server umount lustre-OST0000 complete [ 8519.136744] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8519.146660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8524.256990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8524.259682] Lustre: Skipped 2 previous similar messages [ 8525.258590] Lustre: server umount lustre-MDT0000 complete [ 8529.387115] LustreError: 271377:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8529.408395] LustreError: 271377:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8529.454980] LustreError: 271362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789384103 with bad export cookie 13181182858376019811 [ 8529.486217] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8529.489306] LustreError: 271362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8530.069898] Lustre: server umount lustre-MDT0001 complete [ 8543.172598] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing unload_modules_local [ 8546.227493] Key type lgssc unregistered [ 8546.595047] LNet: 276081:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8546.609322] LNetError: 276081:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8546.628799] LNet: Removed LNI 192.168.202.147@tcp [ 8547.632968] Key type .llcrypt unregistered [ 8547.636421] Key type ._llcrypt unregistered [ 8566.575693] Key type ._llcrypt registered [ 8566.579782] Key type .llcrypt registered [ 8566.745293] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [ 8567.962800] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8568.163076] alg: No test for adler32 (adler32-zlib) [ 8569.258962] Lustre: Lustre: Build Version: 2.17.58_40_gbbf22b4 [ 8569.585652] LNet: Added LNI 192.168.202.147@tcp [8/256/0/180] [ 8571.255255] Key type lgssc registered [ 8572.227783] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8580.314947] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8580.375839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8582.666736] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8586.877512] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 8590.376470] LustreError: 277810:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 8591.842529] LustreError: 277315:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8591.865895] LustreError: 277315:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8592.738917] Lustre: New target registration enabled indefinitely [ 8596.974650] LustreError: 277317:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8601.022517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8601.612437] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8607.126751] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 8614.928551] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8620.499296] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8626.893208] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8627.206616] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8632.307299] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8634.044952] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 8640.630183] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8653.515865] LustreError: 280255:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 8657.826365] Lustre: server umount lustre-OST0000 complete [ 8663.011419] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8663.019309] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8663.032093] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8667.104764] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8667.107106] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8672.226022] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8672.234155] Lustre: Skipped 2 previous similar messages [ 8673.364568] Lustre: server umount lustre-MDT0000 complete [ 8677.306417] LustreError: 277280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789384251 with bad export cookie 17305793448524532776 [ 8677.319861] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8677.325051] LustreError: 277280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8687.793221] Lustre: server umount lustre-MDT0001 complete [ 8698.726722] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing unload_modules_local [ 8701.884642] Key type lgssc unregistered [ 8702.270948] LNet: 281409:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8702.287815] LNetError: 281409:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8702.308074] LNet: Removed LNI 192.168.202.147@tcp [ 8703.249697] Key type .llcrypt unregistered [ 8703.252180] Key type ._llcrypt unregistered [ 8715.912657] Key type ._llcrypt registered [ 8715.914621] Key type .llcrypt registered [ 8716.012689] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8734.221482] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_hostid [ 8750.761323] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [ 8752.013963] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8752.040404] alg: No test for adler32 (adler32-zlib) [ 8753.023303] Lustre: Lustre: Build Version: 2.17.58_40_gbbf22b4 [ 8753.317084] LNet: Added LNI 192.168.202.147@tcp [8/256/0/180] [ 8754.991479] Key type lgssc registered [ 8756.226227] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8791.538062] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8791.584689] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8792.980802] LustreError: 284963:0:(mgs_llog.c:673:mgs_find_or_make_fsdb_nolock()) lustre: New filesystem registration disabled. Use 'lctl set_param allow_register=unlimited' to allow new filesystem registrations: rc = -13 [ 8793.008215] Lustre: First target registration detected: opening a one-time 24-hour registration window. Use 'lctl set_param allow_register=0' to close it early, or 'lctl set_param -P allow_register=0' to lock down permanently. [ 8793.091877] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8793.158511] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8793.280983] Lustre: lustre-MDT0000: new disk, initializing [ 8793.401658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8793.444065] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8797.739496] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 8803.418289] LustreError: 285496:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 8805.299224] Lustre: New target registration enabled indefinitely [ 8813.314775] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8813.443661] Lustre: 284962:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8813.505145] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8813.510717] Lustre: Skipped 1 previous similar message [ 8813.604260] Lustre: lustre-MDT0001: new disk, initializing [ 8813.714234] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8813.809244] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8813.823032] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8820.418706] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 8832.537890] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8840.367996] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8847.981989] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8848.400090] Lustre: lustre-OST0000: new disk, initializing [ 8848.416964] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8848.427334] Lustre: 287174:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8848.606607] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8854.150088] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8854.162189] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8854.315335] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8856.507362] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 8867.624807] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8872.742818] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8873.028839] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8878.124379] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8878.383359] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8885.220018] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8885.228407] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8885.244540] Lustre: Skipped 1 previous similar message [ 8885.245805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8885.256102] Lustre: Skipped 1 previous similar message [ 8886.436218] Lustre: server umount lustre-OST0000 complete [ 8890.336671] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8890.347419] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8895.461287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8895.465499] Lustre: Skipped 2 previous similar messages [ 8896.730285] Lustre: server umount lustre-MDT0000 complete [ 8900.827459] LustreError: 287173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789384474 with bad export cookie 242350562550175861 [ 8900.828371] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8900.835580] LustreError: 287173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8901.305683] Lustre: server umount lustre-MDT0001 complete [ 8909.873261] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 07:14:42 (1789384482) [ 8917.360121] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8917.922087] LustreError: 289588:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8917.946829] LustreError: 289588:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 8918.074142] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8923.107542] LustreError: 289589:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8923.130776] LustreError: 289589:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8923.466683] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 8928.227218] LustreError: 289588:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8933.345873] LustreError: 289589:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8934.411426] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8934.753118] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8939.894799] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 8948.305702] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8955.016056] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8961.846241] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8962.073289] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8968.328686] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 8976.232410] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8989.528306] LustreError: 292188:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 8993.778804] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8993.791692] LustreError: Skipped 1 previous similar message [ 8993.793477] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8993.806211] Lustre: Skipped 1 previous similar message [ 8993.825326] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8994.789735] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8999.767541] Lustre: server umount lustre-OST0000 complete [ 9005.028343] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9005.033905] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9005.042542] Lustre: Skipped 2 previous similar messages [ 9005.059145] Lustre: Skipped 3 previous similar messages [ 9015.266062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9015.270524] Lustre: Skipped 2 previous similar messages [ 9018.335895] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9018.635706] Lustre: server umount lustre-MDT0000 complete [ 9020.403238] LustreError: 289585:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9020.424987] LustreError: 289585:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9022.641871] LustreError: 289568:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789384596 with bad export cookie 242350562550176736 [ 9022.646317] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9022.650685] LustreError: 289568:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9023.020772] Lustre: server umount lustre-MDT0001 complete [ 9035.909330] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing unload_modules_local [ 9039.370221] Key type lgssc unregistered [ 9039.654947] LNet: 293340:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9039.661170] LNetError: 293340:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9040.685316] LNet: Removed LNI 192.168.202.147@tcp [ 9041.712447] Key type .llcrypt unregistered [ 9041.714773] Key type ._llcrypt unregistered [ 9075.809524] Key type ._llcrypt registered [ 9075.812373] Key type .llcrypt registered [ 9075.946512] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_hostid [ 9093.965891] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [ 9095.403262] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9095.612536] alg: No test for adler32 (adler32-zlib) [ 9096.901130] Lustre: Lustre: Build Version: 2.17.58_40_gbbf22b4 [ 9097.319527] LNet: Added LNI 192.168.202.147@tcp [8/256/0/180] [ 9099.111253] Key type lgssc registered [ 9100.779823] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9152.898985] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9152.912257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9154.136776] LustreError: 297168:0:(mgs_llog.c:673:mgs_find_or_make_fsdb_nolock()) lustre: New filesystem registration disabled. Use 'lctl set_param allow_register=unlimited' to allow new filesystem registrations: rc = -13 [ 9154.154979] Lustre: First target registration detected: opening a one-time 24-hour registration window. Use 'lctl set_param allow_register=0' to close it early, or 'lctl set_param -P allow_register=0' to lock down permanently. [ 9154.192027] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9154.220812] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9154.304588] Lustre: lustre-MDT0000: new disk, initializing [ 9154.391406] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9154.419466] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9159.996701] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 9166.777385] LustreError: 297701:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 9168.962563] Lustre: New target registration enabled indefinitely [ 9177.943828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9178.256215] Lustre: 297168:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9178.364698] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9178.377506] Lustre: Skipped 1 previous similar message [ 9178.519221] Lustre: lustre-MDT0001: new disk, initializing [ 9178.701442] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9178.762450] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9178.779380] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9184.743836] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 9195.734414] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9202.795494] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9211.884276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9212.321629] Lustre: lustre-OST2710: new disk, initializing [ 9212.325446] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9212.331211] Lustre: 299380:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9212.432807] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9218.137604] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9218.147482] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9218.279753] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9219.993948] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 9231.653317] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9239.563965] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9239.796940] Lustre: lustre-OST03e8: new disk, initializing [ 9239.801059] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9239.822051] Lustre: 300445:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9239.983541] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9245.382442] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9245.401281] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9245.455874] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9247.486813] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 9255.796628] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9263.567530] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9263.872680] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9269.639264] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9270.071472] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9274.354682] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9274.538848] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9280.524296] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9280.913534] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9291.237459] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9291.240240] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9291.283945] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9291.748799] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9291.791127] Lustre: Skipped 1 previous similar message [ 9291.796244] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9291.798912] Lustre: Skipped 1 previous similar message [ 9293.791120] Lustre: server umount lustre-MDT0000 complete [ 9298.813947] LustreError: 297160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789384872 with bad export cookie 11563300479540569717 [ 9298.816780] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9298.833946] LustreError: 297160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9299.528833] Lustre: server umount lustre-MDT0001 complete [ 9314.770983] Lustre: server umount lustre-OST2710 complete [ 9318.119846] Lustre: 294962:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789384876/real 1789384876] req@0000000096d62920 x1876305820853504/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789384892 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9318.174244] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9318.184635] Lustre: Skipped 1 previous similar message [ 9319.654360] Lustre: server umount lustre-OST03e8 complete [ 9343.032436] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_hostid [ 9355.589737] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [ 9398.868460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9399.110593] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9399.135791] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9399.205453] Lustre: lustre-MDT0000: new disk, initializing [ 9399.268611] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9399.287161] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9404.942509] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 9418.326869] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9418.463163] Lustre: 305699:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9418.506592] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9418.512270] Lustre: Skipped 1 previous similar message [ 9418.570524] Lustre: lustre-MDT0001: new disk, initializing [ 9418.667455] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9418.708465] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9418.731323] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9423.226165] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 9432.905061] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9439.410738] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9447.145034] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9447.582926] Lustre: lustre-OST0000: new disk, initializing [ 9447.590170] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9447.598405] Lustre: 307714:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9447.741209] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9454.690029] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9454.701472] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9454.852444] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9455.323216] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 9465.408340] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9470.649055] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9470.947873] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9476.222302] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9476.520028] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9480.673135] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9480.688067] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9480.707079] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9480.717744] Lustre: Skipped 1 previous similar message [ 9485.037436] Lustre: server umount lustre-OST0000 complete [ 9490.400307] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9490.408357] LustreError: Skipped 1 previous similar message [ 9490.414181] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9490.425240] Lustre: Skipped 1 previous similar message [ 9490.428495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9495.586328] Lustre: server umount lustre-MDT0000 complete [ 9496.038941] LustreError: 305711:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9496.050498] LustreError: 305711:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 9500.145141] LustreError: 305706:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9500.166299] LustreError: 305706:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9500.406371] LustreError: 305690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789385074 with bad export cookie 11563300479540571411 [ 9500.408863] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9500.416600] LustreError: 305690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9500.795146] Lustre: server umount lustre-MDT0001 complete [ 9509.830764] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 07:24:42 (1789385082) [ 9511.500164] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9513.173868] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 07:24:46 (1789385086) [ 9517.624740] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [ 9548.040648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9548.280984] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9548.317491] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9548.477441] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9554.181378] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 9565.098775] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9565.245247] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9565.315885] Lustre: 311571:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9570.997213] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 9593.024467] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9593.390654] Lustre: Denying initial registration attempt from nid 192.168.202.147@tcp, specified as failover [ 9593.400719] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9593.411280] LustreError: 313075:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -99 [ 9593.424370] LustreError: 313075:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 9593.436405] LustreError: 313075:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9593.523694] Lustre: server umount lustre-OST0000 complete [ 9593.527365] LustreError: 313075:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9596.582286] LustreError: 313241:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 9603.556430] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9603.559978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9603.579324] Lustre: Skipped 2 previous similar messages [ 9603.598891] Lustre: Skipped 2 previous similar messages [ 9609.923670] Lustre: server umount lustre-MDT0000 complete [ 9613.793968] LustreError: 311585:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9613.803688] LustreError: 311585:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9613.825491] LustreError: 311564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789385187 with bad export cookie 11563300479540572293 [ 9613.830117] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9613.845844] LustreError: 311564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9614.421241] Lustre: server umount lustre-MDT0001 complete [ 9627.777506] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing unload_modules_local [ 9631.340465] Key type lgssc unregistered [ 9631.618322] LNet: 314342:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9631.625284] LNetError: 314342:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9631.646244] LNet: Removed LNI 192.168.202.147@tcp [ 9632.529686] Key type .llcrypt unregistered [ 9632.531262] Key type ._llcrypt unregistered [ 9645.560333] Key type ._llcrypt registered [ 9645.563815] Key type .llcrypt registered [ 9645.698362] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [ 9646.700248] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9646.865242] alg: No test for adler32 (adler32-zlib) [ 9648.065488] Lustre: Lustre: Build Version: 2.17.58_40_gbbf22b4 [ 9648.327348] LNet: Added LNI 192.168.202.147@tcp [8/256/0/180] [ 9650.023309] Key type lgssc registered [ 9651.434435] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9699.665529] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [ 9709.136965] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9709.167384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9710.546345] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9710.560802] LustreError: 317105:0:(mgs_llog.c:673:mgs_find_or_make_fsdb_nolock()) lustre: New filesystem registration disabled. Use 'lctl set_param allow_register=unlimited' to allow new filesystem registrations: rc = -13 [ 9710.596629] LustreError: 317105:0:(mgs_handler.c:593:mgs_target_reg()) Can't get db for lustre: -13 [ 9710.620105] LustreError: lustre-MDT0000: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 9710.629292] LustreError: 317089:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [ 9710.638172] LustreError: 317089:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 9710.646951] LustreError: 317089:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 9710.754822] Lustre: server umount lustre-MDT0000 complete [ 9710.768925] LustreError: 317089:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 9731.305412] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9792.031307] LustreError: lustre-OST0000: cannot register this server with the MGS: rc = -110. Is the MGS running? [ 9792.041451] LustreError: 317472:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -110 [ 9792.051453] LustreError: 317472:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 9792.059770] LustreError: 317472:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9792.158461] Lustre: server umount lustre-OST0000 complete [ 9792.162557] LustreError: 317472:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -110 [ 9796.193122] Lustre: DEBUG MARKER: conf-sanity test_57b: @@@@@@ FAIL: OST registration from servicenode should not fail [ 9801.958706] Lustre: DEBUG MARKER: == conf-sanity test 58: missing llog files must not prevent MDT from mounting ========================================================== 07:29:34 (1789385374) [ 9809.080305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9809.226678] LustreError: 318218:0:(mgs_llog.c:673:mgs_find_or_make_fsdb_nolock()) lustre: New filesystem registration disabled. Use 'lctl set_param allow_register=unlimited' to allow new filesystem registrations: rc = -13 [ 9809.242946] Lustre: First target registration detected: opening a one-time 24-hour registration window. Use 'lctl set_param allow_register=0' to close it early, or 'lctl set_param -P allow_register=0' to lock down permanently. [ 9809.270324] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9809.307990] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9809.470847] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9814.654761] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 9819.360642] LustreError: 318700:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 9821.464819] Lustre: New target registration enabled indefinitely [ 9829.085635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9829.180595] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9829.237534] Lustre: 318217:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9829.468843] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9834.027622] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 9841.166444] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9846.609431] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9854.449023] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9854.616971] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9854.773948] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9861.269736] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 9868.818553] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9876.961214] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9876.971700] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9876.979015] Lustre: Skipped 1 previous similar message [ 9876.991438] Lustre: Skipped 2 previous similar messages [ 9880.826449] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.47@tcp (stopping) [ 9882.080551] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9882.084216] Lustre: Skipped 2 previous similar messages [ 9882.309921] Lustre: server umount lustre-MDT0000 complete [ 9885.393070] LustreError: 318210:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789385459 with bad export cookie 9881524592940475064 [ 9885.401291] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9885.406781] LustreError: 318210:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9885.979587] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.47@tcp (stopping) [ 9887.208175] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9887.215968] Lustre: Skipped 1 previous similar message [ 9890.214769] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9890.232228] Lustre: Skipped 1 previous similar message [ 9891.055661] LustreError: 318224:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9891.077146] LustreError: 318224:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 9892.132543] Lustre: server umount lustre-MDT0001 complete [ 9899.970225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9909.737601] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9910.984902] LustreError: 320318:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9911.083697] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9911.166514] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 9915.181073] LustreError: 320319:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9915.210230] LustreError: 320319:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9915.533440] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 9920.306891] LustreError: 320318:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9920.318110] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9920.324249] LustreError: 320318:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9925.614697] LustreError: 322091:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9925.640674] LustreError: 322091:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9925.716506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9925.870711] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.47@tcp (not set up) [ 9925.879660] Lustre: Skipped 1 previous similar message [ 9926.033732] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9930.518772] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [ 9931.250978] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9937.313101] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9939.412753] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9946.193456] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9948.063685] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9952.994417] LustreError: 323811:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 9957.345494] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9957.360379] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9957.379929] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9961.961534] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9961.972151] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9963.529384] Lustre: server umount lustre-OST0000 complete [ 9972.193372] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9972.194784] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9972.215967] Lustre: Skipped 1 previous similar message [ 9973.834231] Lustre: server umount lustre-MDT0000 complete [ 9977.312468] LustreError: 322098:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9977.334992] LustreError: 322098:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9978.183720] LustreError: 318208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789385552 with bad export cookie 9881524592940482925 [ 9978.197236] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9978.204303] LustreError: 318208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9978.735996] Lustre: server umount lustre-MDT0001 complete [ 9995.650149] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing unload_modules_local [10001.099615] Key type lgssc unregistered [10001.544952] LNet: 324963:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10001.551937] LNetError: 324963:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10001.578830] LNet: Removed LNI 192.168.202.147@tcp [10002.843577] Key type .llcrypt unregistered [10002.848132] Key type ._llcrypt unregistered [10030.690734] Key type ._llcrypt registered [10030.693615] Key type .llcrypt registered [10030.792816] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [10031.630923] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10031.725599] alg: No test for adler32 (adler32-zlib) [10032.692571] Lustre: Lustre: Build Version: 2.17.58_40_gbbf22b4 [10032.999698] LNet: Added LNI 192.168.202.147@tcp [8/256/0/180] [10034.736146] Key type lgssc registered [10036.069944] Lustre: Echo OBD driver; http://www.lustre.org/ [10043.940778] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10043.953783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10045.611930] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10049.932641] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [10053.308901] LustreError: 326849:0:(llog.c:449:llog_init_handle()) llog type is not specified! [10055.150554] LustreError: 326380:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10055.172275] LustreError: 326380:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10055.223509] Lustre: New target registration enabled indefinitely [10060.256950] LustreError: 326379:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10063.567263] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10063.916354] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10068.715939] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [10075.926204] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10082.115297] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10089.442029] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10089.443427] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10089.474377] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10089.505925] Lustre: Skipped 1 previous similar message [10090.528342] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10090.535394] Lustre: Skipped 1 previous similar message [10092.180897] Lustre: server umount lustre-MDT0000 complete [10095.584601] LustreError: 327350:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10095.600496] LustreError: 327350:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10095.990247] LustreError: 326360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789385670 with bad export cookie 12800558673098781153 [10095.998401] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10096.005797] LustreError: 326360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10096.637174] Lustre: server umount lustre-MDT0001 complete [10111.561830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10111.910266] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10111.960990] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10112.309474] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10117.142484] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [10126.989098] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10127.119929] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10127.168899] Lustre: 329210:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10127.488366] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10133.022955] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [10140.997749] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10147.981901] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10158.529089] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10158.883376] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10158.900267] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10158.916487] LustreError: 331143:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [10158.932848] LustreError: 331143:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [10158.947723] LustreError: 331143:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10159.060254] Lustre: server umount lustre-OST0000 complete [10159.063516] LustreError: 331143:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10171.443711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10171.794903] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10172.075198] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10178.835477] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [10179.687612] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [10188.288253] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10197.286907] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10197.397490] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10197.523675] Lustre: lustre-OST0001: new disk, initializing [10197.529811] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10197.539347] Lustre: 332686:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10197.671722] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10204.790437] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [10205.315206] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10205.339864] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10205.536602] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10213.536592] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10221.024521] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10221.029929] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10221.038132] LustreError: Skipped 1 previous similar message [10221.068129] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10221.070553] Lustre: Skipped 2 previous similar messages [10221.095070] Lustre: Skipped 1 previous similar message [10222.796467] Lustre: server umount lustre-OST0001 complete [10226.167538] LustreError: 331610:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10226.190150] LustreError: 331610:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10226.259495] LustreError: 333641:0:(llog.c:449:llog_init_handle()) llog type is not specified! [10231.266430] LustreError: 331929:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10231.267612] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10231.277262] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10231.297541] LustreError: 331929:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10231.318302] Lustre: Skipped 1 previous similar message [10236.393828] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10236.415836] Lustre: Skipped 2 previous similar messages [10237.312863] Lustre: server umount lustre-OST0000 complete [10245.088652] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10245.104709] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10245.130822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10257.379478] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10257.483832] Lustre: server umount lustre-MDT0000 complete [10260.961279] LustreError: 329221:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10260.987147] LustreError: 329221:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10262.624491] LustreError: 329201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789385836 with bad export cookie 12800558673098781699 [10262.627609] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10262.647608] LustreError: 329201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10263.562540] Lustre: server umount lustre-MDT0001 complete [10281.058268] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing unload_modules_local [10284.709873] Key type lgssc unregistered [10285.044786] LNet: 334812:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10285.052871] LNetError: 334812:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10285.073615] LNet: Removed LNI 192.168.202.147@tcp [10286.043988] Key type .llcrypt unregistered [10286.046096] Key type ._llcrypt unregistered [10365.867934] Key type ._llcrypt registered [10365.870898] Key type .llcrypt registered [10366.021638] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_hostid [10384.129508] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [10384.875891] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10384.929306] alg: No test for adler32 (adler32-zlib) [10385.974573] Lustre: Lustre: Build Version: 2.17.58_40_gbbf22b4 [10386.427584] LNet: Added LNI 192.168.202.147@tcp [8/256/0/180] [10388.119237] Key type lgssc registered [10389.905758] Lustre: Echo OBD driver; http://www.lustre.org/ [10433.810451] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10433.840926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10435.236191] LustreError: 339748:0:(mgs_llog.c:673:mgs_find_or_make_fsdb_nolock()) lustre: New filesystem registration disabled. Use 'lctl set_param allow_register=unlimited' to allow new filesystem registrations: rc = -13 [10435.250603] Lustre: First target registration detected: opening a one-time 24-hour registration window. Use 'lctl set_param allow_register=0' to close it early, or 'lctl set_param -P allow_register=0' to lock down permanently. [10435.307551] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10435.343567] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10435.428060] Lustre: lustre-MDT0000: new disk, initializing [10435.558582] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10435.604690] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10441.952993] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [10448.504923] LustreError: 340281:0:(llog.c:449:llog_init_handle()) llog type is not specified! [10450.720086] Lustre: New target registration enabled indefinitely [10458.772459] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10459.016410] Lustre: 339745:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10459.166375] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10459.178995] Lustre: Skipped 1 previous similar message [10459.304618] Lustre: lustre-MDT0001: new disk, initializing [10459.440708] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10459.491685] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10459.505509] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10465.109826] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [10476.218048] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10483.573723] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10491.390760] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10491.820381] Lustre: lustre-OST0000: new disk, initializing [10491.823672] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10491.831201] Lustre: 341959:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10491.992882] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10499.331814] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [10499.655257] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10499.672223] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10499.822063] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10509.707840] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10515.072139] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10515.319701] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10520.536529] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10520.783218] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10525.665180] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10525.676932] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10525.681641] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10525.693050] Lustre: Skipped 1 previous similar message [10525.706321] Lustre: Skipped 1 previous similar message [10529.448450] Lustre: server umount lustre-OST0000 complete [10535.905152] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10535.915035] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10535.919026] Lustre: Skipped 1 previous similar message [10535.936268] Lustre: Skipped 1 previous similar message [10540.672473] Lustre: server umount lustre-MDT0000 complete [10545.278040] LustreError: 339740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789386119 with bad export cookie 16359851075496714317 [10545.291787] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10545.292887] LustreError: 339740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10545.746082] Lustre: server umount lustre-MDT0001 complete [10556.003662] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 07:42:08 (1789386128) [10565.421338] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 07:42:17 (1789386137) [10576.180353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10576.820887] LustreError: 344716:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10576.855911] LustreError: 344716:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [10576.934516] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10581.993812] LustreError: 344717:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10582.480337] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [10587.106458] LustreError: 344716:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10592.229101] LustreError: 344717:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10595.402959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10596.420473] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10603.754504] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [10614.562772] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10621.352956] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10631.157565] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10631.623322] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10639.478752] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [10648.179647] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10657.837843] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10659.895259] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10662.409738] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10664.851660] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10667.531172] Lustre: Failing over lustre-MDT0000 [10668.003844] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10668.019380] LustreError: Skipped 1 previous similar message [10668.026068] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10668.040046] LustreError: 344716:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10668.065469] Lustre: server umount lustre-MDT0000 complete [10672.616837] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10676.977810] LustreError: 345696:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10677.016518] LustreError: 345696:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [10687.903420] Lustre: 337970:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789386246/real 1789386246] req@000000009498fdfa x1876307172470656/t0(0) o400->MGC192.168.202.147@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789386262 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10687.939498] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10688.167877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10693.092319] LustreError: 345493:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10693.128383] LustreError: 345493:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 13 previous similar messages [10698.220320] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xe309dddffd4627d2 [10698.229429] Lustre: MGC192.168.202.147@tcp: Connection restored to 0@lo (at 0@lo) [10698.452028] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10698.520444] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10703.644696] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10703.877327] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10703.949506] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [10704.013958] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10704.141981] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [10715.566674] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10717.347520] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10721.457173] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10723.981986] Lustre: Failing over lustre-MDT0000 [10724.105556] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.47@tcp (stopping) [10724.315538] Lustre: server umount lustre-MDT0000 complete [10724.338184] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10724.352224] Lustre: Skipped 3 previous similar messages [10729.214523] LustreError: 345696:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10729.233491] LustreError: 345696:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [10740.713642] Lustre: 337969:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789386298/real 1789386298] req@00000000fa9f9226 x1876307172491520/t0(0) o400->MGC192.168.202.147@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789386314 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10740.745895] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10744.486867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10749.921391] LustreError: 337966:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000a387263d x1876307172497024/t0(0) o250->MGC192.168.202.147@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10750.398874] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10750.448558] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10750.710993] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10755.356988] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [10755.580952] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10755.605062] Lustre: Skipped 2 previous similar messages [10755.655311] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [10755.709519] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10765.729275] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10767.246545] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10776.036292] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10776.039834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10776.062844] Lustre: Skipped 2 previous similar messages [10776.081638] Lustre: Skipped 2 previous similar messages [10780.302953] Lustre: server umount lustre-MDT0000 complete [10784.409214] LustreError: 345500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789386358 with bad export cookie 16359851075496717040 [10784.425515] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10784.426256] LustreError: 345500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10784.843219] Lustre: server umount lustre-MDT0001 complete [10800.026251] Lustre: server umount lustre-OST0000 complete [10817.103296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10817.768771] LustreError: 351110:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10817.791790] LustreError: 351110:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 25 previous similar messages [10817.890871] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10823.084601] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [10833.903286] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10839.249968] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [10847.530392] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10853.385294] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10861.100117] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10862.582413] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10868.375874] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [10876.396249] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10885.446900] LustreError: 353689:0:(llog.c:449:llog_init_handle()) llog type is not specified! [10889.187752] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10889.214248] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10891.473617] Lustre: server umount lustre-OST0000 complete [10895.842574] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10899.425368] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10899.432618] Lustre: Skipped 3 previous similar messages [10910.175192] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10910.618626] Lustre: server umount lustre-MDT0000 complete [10914.433799] LustreError: 351092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789386488 with bad export cookie 16359851075496717782 [10914.442367] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10914.453919] LustreError: 351092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10915.330114] Lustre: server umount lustre-MDT0001 complete [10927.654245] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing unload_modules_local [10931.447213] Key type lgssc unregistered [10931.827435] LNet: 354840:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10931.833978] LNetError: 354840:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10931.859640] LNet: Removed LNI 192.168.202.147@tcp [10933.033392] Key type .llcrypt unregistered [10933.035767] Key type ._llcrypt unregistered [10964.659590] Key type ._llcrypt registered [10964.662253] Key type .llcrypt registered [10964.786271] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [10966.401945] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10966.481647] alg: No test for adler32 (adler32-zlib) [10967.633305] Lustre: Lustre: Build Version: 2.17.58_40_gbbf22b4 [10967.987967] LNet: Added LNI 192.168.202.147@tcp [8/256/0/180] [10969.863235] Key type lgssc registered [10971.408576] Lustre: Echo OBD driver; http://www.lustre.org/ [10981.515712] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10981.568674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10983.460812] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10991.348588] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [10993.642636] LustreError: 356357:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10993.671208] LustreError: 356357:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10996.974945] LustreError: 356830:0:(llog.c:449:llog_init_handle()) llog type is not specified! [10998.754077] LustreError: 356376:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10999.799433] Lustre: New target registration enabled indefinitely [11003.873740] LustreError: 356357:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11007.971431] LustreError: 356376:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11009.085325] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11009.430571] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11014.478737] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [11023.974177] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11031.364438] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11039.994723] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11040.693331] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11047.902744] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [11052.987340] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11056.547300] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11069.009165] Lustre: server umount lustre-MDT0000 complete [11069.418568] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11069.421191] LustreError: 356352:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11069.435435] Lustre: Skipped 1 previous similar message [11069.465444] LustreError: 356352:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11073.683562] LustreError: 356336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789386647 with bad export cookie 16802435547215435940 [11073.690790] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11073.696471] LustreError: 356336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11073.955208] Lustre: server umount lustre-MDT0001 complete [11087.132684] Lustre: server umount lustre-OST0000 complete [11102.023850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11102.674478] LustreError: 360315:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11102.769084] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11107.974610] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [11119.253095] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11119.937380] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11127.268073] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [11137.212566] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11145.361129] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11153.412778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11153.958060] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11161.559967] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [11169.513872] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11172.433830] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11191.785115] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11191.798200] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11191.812922] Lustre: Skipped 1 previous similar message [11191.828738] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11192.800757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11192.815614] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11192.819250] Lustre: Skipped 1 previous similar message [11192.847287] Lustre: Skipped 1 previous similar message [11194.687882] Lustre: server umount lustre-MDT0000 complete [11197.925395] LustreError: 360316:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11197.955242] LustreError: 360316:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [11198.484713] LustreError: 360297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789386772 with bad export cookie 16802435547215437053 [11198.493126] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11198.498234] LustreError: 360297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11198.918508] Lustre: server umount lustre-MDT0001 complete [11213.909867] Lustre: server umount lustre-OST0000 complete [11234.286273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11234.829745] LustreError: 364278:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11234.927685] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11239.085301] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [11251.217328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11257.229838] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [11266.340653] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11272.362909] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11279.553151] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11279.918713] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11279.925753] Lustre: Skipped 1 previous similar message [11280.943082] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11286.921905] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [11295.438744] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11300.496716] LustreError: 366857:0:(llog.c:449:llog_init_handle()) llog type is not specified! [11306.468887] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11306.472310] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11306.494431] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11306.499117] LustreError: Skipped 1 previous similar message [11306.503078] Lustre: Skipped 1 previous similar message [11311.588957] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11311.599549] Lustre: Skipped 1 previous similar message [11316.714319] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11316.724601] Lustre: Skipped 1 previous similar message [11320.287370] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11320.487683] Lustre: server umount lustre-OST0000 complete [11326.946661] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11326.948935] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11326.971930] Lustre: Skipped 2 previous similar messages [11326.984565] Lustre: Skipped 2 previous similar messages [11330.943592] Lustre: server umount lustre-MDT0000 complete [11332.068176] LustreError: 364273:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11332.084368] LustreError: 364273:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [11334.400830] LustreError: 364260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789386908 with bad export cookie 16802435547215438075 [11334.407826] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11334.421115] LustreError: 364260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11334.813131] Lustre: server umount lustre-MDT0001 complete [11347.335413] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing unload_modules_local [11349.811751] Key type lgssc unregistered [11350.161277] LNet: 368009:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11350.167219] LNetError: 368009:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11350.191684] LNet: Removed LNI 192.168.202.147@tcp [11350.732193] Key type .llcrypt unregistered [11350.733548] Key type ._llcrypt unregistered [11381.627564] Key type ._llcrypt registered [11381.631155] Key type .llcrypt registered [11381.775170] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [11384.000099] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11384.255362] alg: No test for adler32 (adler32-zlib) [11385.475364] Lustre: Lustre: Build Version: 2.17.58_40_gbbf22b4 [11385.852959] LNet: Added LNI 192.168.202.147@tcp [8/256/0/180] [11387.743428] Key type lgssc registered [11389.852444] Lustre: Echo OBD driver; http://www.lustre.org/ [11399.450895] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11399.469947] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11399.475609] LustreError: 369447:0:(osd_handler.c:8546:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11399.494519] LustreError: 369447:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11399.500363] LustreError: 369447:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11399.512336] LustreError: 369447:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11399.522578] LustreError: 369447:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11409.574483] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11409.589315] LustreError: 369844:0:(osd_handler.c:8546:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11409.626964] LustreError: 369844:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11409.634710] LustreError: 369844:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11409.643878] LustreError: 369844:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11409.663126] LustreError: 369844:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11415.036913] Lustre: New target registration enabled indefinitely [11435.551848] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing unload_modules_local [11439.059919] Key type lgssc unregistered [11439.314410] LNet: 370995:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11439.320133] LNetError: 370995:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11439.343484] LNet: Removed LNI 192.168.202.147@tcp [11440.261385] Key type .llcrypt unregistered [11440.267868] Key type ._llcrypt unregistered [11459.812451] Key type ._llcrypt registered [11459.814481] Key type .llcrypt registered [11459.993736] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_hostid [11475.031365] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [11476.435380] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11476.655945] alg: No test for adler32 (adler32-zlib) [11478.021459] Lustre: Lustre: Build Version: 2.17.58_40_gbbf22b4 [11478.589053] LNet: Added LNI 192.168.202.147@tcp [8/256/0/180] [11480.487198] Key type lgssc registered [11482.619690] Lustre: Echo OBD driver; http://www.lustre.org/ [11524.160200] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11524.218097] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11525.589012] LustreError: 374152:0:(mgs_llog.c:673:mgs_find_or_make_fsdb_nolock()) lustre: New filesystem registration disabled. Use 'lctl set_param allow_register=unlimited' to allow new filesystem registrations: rc = -13 [11525.600543] Lustre: First target registration detected: opening a one-time 24-hour registration window. Use 'lctl set_param allow_register=0' to close it early, or 'lctl set_param -P allow_register=0' to lock down permanently. [11525.619925] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11525.649561] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11525.726736] Lustre: lustre-MDT0000: new disk, initializing [11525.830761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11525.848308] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11531.903990] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [11539.788644] LustreError: 374685:0:(llog.c:449:llog_init_handle()) llog type is not specified! [11542.687388] Lustre: New target registration enabled indefinitely [11554.183969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11554.296853] Lustre: 374151:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11554.377620] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11554.382532] Lustre: Skipped 1 previous similar message [11554.460449] Lustre: lustre-MDT0001: new disk, initializing [11554.571984] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11554.599151] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11554.609743] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11559.516595] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [11569.967982] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11576.078658] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11582.649071] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11582.954475] Lustre: lustre-OST0000: new disk, initializing [11582.960349] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11582.966628] Lustre: 376364:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11583.068015] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11589.240225] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11589.254187] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11589.393821] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11590.634202] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [11601.692289] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11608.125708] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11608.358258] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11612.800988] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11612.983629] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11615.215955] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11615.216755] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11615.235095] LustreError: Skipped 1 previous similar message [11615.250217] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11620.336859] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11620.346834] Lustre: Skipped 2 previous similar messages [11625.450828] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11629.023956] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11629.270888] Lustre: server umount lustre-OST0000 complete [11635.681629] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11635.703054] Lustre: Skipped 1 previous similar message [11635.715777] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11635.729078] Lustre: Skipped 2 previous similar messages [11640.129383] Lustre: server umount lustre-MDT0000 complete [11644.494204] LustreError: 374142:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789387218 with bad export cookie 12336627139982653280 [11644.499256] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11644.515812] LustreError: 374142:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11645.114768] Lustre: server umount lustre-MDT0001 complete [11656.422746] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11658.520505] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 08:00:30 (1789387230) [11664.760080] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11665.129142] LustreError: 378828:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11665.153672] LustreError: 378828:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11665.219740] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11670.023655] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [11670.498346] LustreError: 378829:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11675.625502] LustreError: 378828:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11680.564404] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11680.745742] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [11680.988171] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11686.931399] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [11696.912903] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11703.464448] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11710.146809] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11710.550325] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11717.239503] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [11725.123526] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11732.123748] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11732.330808] Lustre: lustre-OST0001: new disk, initializing [11732.338288] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11732.344096] Lustre: 381723:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11732.500373] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11739.396645] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11739.428429] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11739.611737] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11739.941898] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [11749.892480] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11754.977487] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11754.981266] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11754.983023] LustreError: Skipped 1 previous similar message [11754.990126] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11755.019111] Lustre: Skipped 2 previous similar messages [11755.037838] Lustre: Skipped 1 previous similar message [11759.901040] Lustre: server umount lustre-OST0001 complete [11761.913232] LustreError: 380711:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11768.300813] LustreError: 380711:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11768.328503] LustreError: 380711:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11776.224474] LustreError: 382708:0:(llog.c:449:llog_init_handle()) llog type is not specified! [11778.532844] LustreError: 380709:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11778.560239] LustreError: 380709:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11781.601678] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11781.612921] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11781.640278] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11781.650231] Lustre: Skipped 3 previous similar messages [11787.290711] Lustre: server umount lustre-OST0000 complete [11793.888331] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11793.891698] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11793.899654] LustreError: Skipped 1 previous similar message [11793.931687] Lustre: Skipped 2 previous similar messages [11797.715623] Lustre: server umount lustre-MDT0000 complete [11799.010077] LustreError: 378829:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11799.025803] LustreError: 378829:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11801.851890] LustreError: 378810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789387376 with bad export cookie 12336627139982654162 [11801.857205] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11801.870423] LustreError: 378810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11808.363755] Lustre: server umount lustre-MDT0001 complete [11820.807810] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing unload_modules_local [11823.947867] Key type lgssc unregistered [11824.316958] LNet: 383861:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11824.323247] LNetError: 383861:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11824.336614] LNet: Removed LNI 192.168.202.147@tcp [11825.162064] Key type .llcrypt unregistered [11825.169248] Key type ._llcrypt unregistered [11878.067967] Key type ._llcrypt registered [11878.070881] Key type .llcrypt registered [11878.183827] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [11879.530884] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11879.602927] alg: No test for adler32 (adler32-zlib) [11880.760253] Lustre: Lustre: Build Version: 2.17.58_40_gbbf22b4 [11881.065746] LNet: Added LNI 192.168.202.147@tcp [8/256/0/180] [11882.823292] Key type lgssc registered [11884.236905] Lustre: Echo OBD driver; http://www.lustre.org/ [11892.482720] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11892.515486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11893.917966] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11893.930865] LustreError: 386663:0:(mgs_llog.c:673:mgs_find_or_make_fsdb_nolock()) lustre: New filesystem registration disabled. Use 'lctl set_param allow_register=unlimited' to allow new filesystem registrations: rc = -13 [11893.952017] LustreError: 386663:0:(mgs_handler.c:593:mgs_target_reg()) Can't get db for lustre: -13 [11893.978879] LustreError: lustre-MDT0000: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11893.991374] LustreError: 386646:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [11893.999521] LustreError: 386646:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [11894.008182] LustreError: 386646:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [11894.124400] Lustre: server umount lustre-MDT0000 complete [11894.126353] LustreError: 386646:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11895.901873] Lustre: DEBUG MARKER: conf-sanity test_65: @@@@@@ FAIL: fail to restart the MDS [11902.385318] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 08:04:34 (1789387474) [11908.402964] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11908.698370] LustreError: 387291:0:(mgs_llog.c:673:mgs_find_or_make_fsdb_nolock()) lustre: New filesystem registration disabled. Use 'lctl set_param allow_register=unlimited' to allow new filesystem registrations: rc = -13 [11908.712265] Lustre: First target registration detected: opening a one-time 24-hour registration window. Use 'lctl set_param allow_register=0' to close it early, or 'lctl set_param -P allow_register=0' to lock down permanently. [11908.725756] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11908.742851] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11909.025638] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11913.896187] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [11918.237755] LustreError: 387774:0:(llog.c:449:llog_init_handle()) llog type is not specified! [11920.400484] Lustre: New target registration enabled indefinitely [11928.775594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11928.978791] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11929.076217] Lustre: 387290:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11929.436642] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11935.840799] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [11944.013190] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11950.700505] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11958.383553] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11958.558575] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11958.743135] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11965.413100] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [11972.993810] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11981.352978] Lustre: Permanently deactivating lustre-OST0000 [11981.375333] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11990.068823] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11992.991308] LustreError: 390596:0:(mgs_llog.c:2046:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11993.002117] LustreError: 390596:0:(mgs_llog.c:2101:mgs_replace_nids()) Only MGS is allowed to be started [11993.008150] LustreError: 390596:0:(mgs_handler.c:1256:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11995.912806] LustreError: 390645:0:(mgs_llog.c:2101:mgs_replace_nids()) Only MGS is allowed to be started [11995.917454] LustreError: 390645:0:(mgs_handler.c:1256:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12004.088757] Lustre: server umount lustre-OST0000 complete [12006.870833] LustreError: 390894:0:(mgs_llog.c:2101:mgs_replace_nids()) Only MGS is allowed to be started [12006.875732] LustreError: 390894:0:(mgs_handler.c:1256:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12008.928532] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12008.931485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12008.949932] Lustre: Skipped 1 previous similar message [12008.972343] Lustre: Skipped 1 previous similar message [12014.053535] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12014.062259] Lustre: Skipped 1 previous similar message [12019.169536] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12019.188094] Lustre: Skipped 1 previous similar message [12023.263575] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12023.996586] Lustre: server umount lustre-MDT0000 complete [12029.077692] LustreError: 387283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789387603 with bad export cookie 849970515529366388 [12029.100844] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12039.835393] Lustre: server umount lustre-MDT0001 complete [12049.933713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12055.258930] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [12059.894507] LustreError: 392103:0:(mgs_handler.c:1256:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12068.163560] Lustre: 392442:0:(mgs_llog.c:1811:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.147@tcp [12070.430496] Lustre: server umount MGS complete [12081.678156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12082.217088] LustreError: 393166:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12082.236216] LustreError: 393166:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12082.244419] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12082.249706] Lustre: Skipped 1 previous similar message [12082.298406] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12087.003860] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [12087.797389] LustreError: 393167:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12092.901968] LustreError: 393166:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12098.021847] LustreError: 393167:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12099.617399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12100.045869] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12100.134122] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12105.717677] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug -1 all [12114.274333] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12120.230847] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12122.116740] Lustre: Permanently reactivating lustre-OST0000 [12122.134269] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12122.138830] Lustre: Skipped 2 previous similar messages [12149.729737] LustreError: 394997:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12157.705874] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12157.980104] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180