[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 487078024 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2878856K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.003340] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005013] kvm-guest: setup PV IPIs [ 0.008548] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010011] pid_max: default: 32768 minimum: 301 [ 0.011125] LSM: Security Framework initializing [ 0.012051] Yama: becoming mindful. [ 0.013034] SELinux: Initializing. [ 0.014063] *** VALIDATE selinux *** [ 0.021746] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025730] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026140] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027113] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028105] *** VALIDATE tmpfs *** [ 0.029374] *** VALIDATE proc *** [ 0.031063] *** VALIDATE cgroup *** [ 0.032009] *** VALIDATE cgroup2 *** [ 0.033244] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035028] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038029] Spectre V2 : User space: Vulnerable [ 0.039012] Speculative Store Bypass: Vulnerable [ 0.042501] debug: unmapping init [mem 0xffffffff85059000-0xffffffff85060fff] [ 0.044169] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045673] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046028] ... version: 2 [ 0.047012] ... bit width: 48 [ 0.048014] ... generic registers: 4 [ 0.049014] ... value mask: 0000ffffffffffff [ 0.050014] ... max period: 00007fffffffffff [ 0.051015] ... fixed-purpose events: 3 [ 0.052014] ... event mask: 000000070000000f [ 0.053294] rcu: Hierarchical SRCU implementation. [ 0.055381] smp: Bringing up secondary CPUs ... [ 0.056606] x86: Booting SMP configuration: [ 0.057029] .... node #0, CPUs: #1 #2 #3 [ 0.060515] smp: Brought up 1 node, 4 CPUs [ 0.062012] smpboot: Max logical packages: 1 [ 0.063016] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.129248] node 0 deferred pages initialised in 64ms [ 0.132162] devtmpfs: initialized [ 0.133236] x86/mm: Memory block size: 128MB [ 0.135717] gcov: version magic: 0x41383552 [ 0.137098] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.138087] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.139233] pinctrl core: initialized pinctrl subsystem [ 0.140201] [ 0.140746] ************************************************************* [ 0.141013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.142014] ** ** [ 0.143014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.144011] ** ** [ 0.145013] ** This means that this kernel is built to expose internal ** [ 0.146008] ** IOMMU data structures, which may compromise security on ** [ 0.147007] ** your system. ** [ 0.148013] ** ** [ 0.149011] ** If you see this message and you are not debugging the ** [ 0.150014] ** kernel, report this immediately to your vendor! ** [ 0.151012] ** ** [ 0.152015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.153011] ************************************************************* [ 0.154710] NET: Registered protocol family 16 [ 0.155351] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.156062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.157062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.159009] cpuidle: using governor menu [ 0.160674] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.163628] PCI: Using configuration type 1 for base access [ 0.166107] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.174294] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.177041] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.180198] cryptd: max_cpu_qlen set to 1000 [ 0.182284] ACPI: Added _OSI(Module Device) [ 0.184017] ACPI: Added _OSI(Processor Device) [ 0.186013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.188011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.193154] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.199346] ACPI: Interpreter enabled [ 0.200056] ACPI: PM: (supports S0 S3 S4 S5) [ 0.202013] ACPI: Using IOAPIC for interrupt routing [ 0.203122] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.206413] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.216048] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.218042] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.221019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.223073] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.228021] acpiphp: Slot [2] registered [ 0.229097] acpiphp: Slot [5] registered [ 0.231120] acpiphp: Slot [6] registered [ 0.232100] acpiphp: Slot [7] registered [ 0.233103] acpiphp: Slot [8] registered [ 0.234051] acpiphp: Slot [9] registered [ 0.236112] acpiphp: Slot [10] registered [ 0.237093] acpiphp: Slot [3] registered [ 0.239093] acpiphp: Slot [4] registered [ 0.240088] acpiphp: Slot [11] registered [ 0.241058] acpiphp: Slot [12] registered [ 0.242076] acpiphp: Slot [13] registered [ 0.243064] acpiphp: Slot [14] registered [ 0.244038] acpiphp: Slot [15] registered [ 0.245087] acpiphp: Slot [16] registered [ 0.246218] acpiphp: Slot [17] registered [ 0.248101] acpiphp: Slot [18] registered [ 0.250100] acpiphp: Slot [19] registered [ 0.251110] acpiphp: Slot [20] registered [ 0.252087] acpiphp: Slot [21] registered [ 0.254101] acpiphp: Slot [22] registered [ 0.255064] acpiphp: Slot [23] registered [ 0.256088] acpiphp: Slot [24] registered [ 0.258091] acpiphp: Slot [25] registered [ 0.260116] acpiphp: Slot [26] registered [ 0.261108] acpiphp: Slot [27] registered [ 0.263115] acpiphp: Slot [28] registered [ 0.265104] acpiphp: Slot [29] registered [ 0.266089] acpiphp: Slot [30] registered [ 0.268102] acpiphp: Slot [31] registered [ 0.269060] PCI host bridge to bus 0000:00 [ 0.271019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.273018] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.275021] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.277029] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.279020] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.280019] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.282185] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.284905] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.287866] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.296013] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.300146] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.303015] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.305016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.307017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.309558] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.312704] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.315031] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.318675] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.321851] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.332748] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.337012] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.342269] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.350016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.357015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.373014] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.383095] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.392011] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.399013] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.423017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.435997] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.443014] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.449011] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.466015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.478545] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.484014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.492015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.517019] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.528900] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.534013] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.539013] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.552015] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.561899] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.569014] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.575013] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.596014] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.608032] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.611435] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.615372] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.617374] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.620248] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.626102] iommu: Default domain type: Passthrough [ 0.627457] SCSI subsystem initialized [ 0.628146] ACPI: bus type USB registered [ 0.629091] usbcore: registered new interface driver usbfs [ 0.631060] usbcore: registered new interface driver hub [ 0.632063] usbcore: registered new device driver usb [ 0.634135] pps_core: LinuxPPS API ver. 1 registered [ 0.635007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.639050] PTP clock support registered [ 0.641007] EDAC MC: Ver: 3.0.0 [ 0.643139] PCI: Using ACPI for IRQ routing [ 0.646061] NetLabel: Initializing [ 0.647011] NetLabel: domain hash size = 128 [ 0.649013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.651097] NetLabel: unlabeled traffic allowed by default [ 0.654039] vgaarb: loaded [ 0.655227] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.657015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.667579] clocksource: Switched to clocksource kvm-clock [ 0.777434] VFS: Disk quotas dquot_6.6.0 [ 0.778915] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.781121] *** VALIDATE ramfs *** [ 0.782290] *** VALIDATE hugetlbfs *** [ 0.784259] pnp: PnP ACPI init [ 0.786399] pnp: PnP ACPI: found 6 devices [ 0.806224] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.809579] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.811593] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.813630] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.815812] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.818157] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.820954] NET: Registered protocol family 2 [ 0.823572] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.828505] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.832117] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.837494] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.841455] TCP: Hash tables configured (established 65536 bind 65536) [ 0.844591] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.847613] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.850695] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.854068] NET: Registered protocol family 1 [ 0.856565] RPC: Registered named UNIX socket transport module. [ 0.859063] RPC: Registered udp transport module. [ 0.861042] RPC: Registered tcp transport module. [ 0.863009] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.865739] NET: Registered protocol family 44 [ 0.867559] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.870831] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.872784] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.875227] PCI: CLS 0 bytes, default 64 [ 0.876913] Unpacking initramfs... [ 2.266159] debug: unmapping init [mem 0xffff9f787cc54000-0xffff9f787ffbffff] [ 2.273676] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.277295] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.281608] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.788702] Initialise system trusted keyrings [ 2.790339] Key type blacklist registered [ 2.793127] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.803927] zbud: loaded [ 2.807618] *** VALIDATE nfs *** [ 2.809177] *** VALIDATE nfs4 *** [ 2.811136] pstore: using deflate compression [ 2.814979] Platform Keyring initialized [ 2.919306] NET: Registered protocol family 38 [ 2.921687] Key type asymmetric registered [ 2.923188] Asymmetric key parser 'x509' registered [ 2.925405] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.928848] io scheduler mq-deadline registered [ 2.930865] io scheduler kyber registered [ 2.932374] io scheduler bfq registered [ 2.934491] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.937619] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.940640] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.943778] ACPI: Power Button [PWRF] [ 2.949354] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.956173] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.974463] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.982821] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.001356] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.031545] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.061308] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.067485] Non-volatile memory driver v1.3 [ 3.069915] Linux agpgart interface v0.103 [ 3.107349] virtio_blk virtio1: [vda] 146704 512-byte logical blocks (75.1 MB/71.6 MiB) [ 3.110509] vda: detected capacity change from 0 to 75112448 [ 3.127377] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.130311] vdb: detected capacity change from 0 to 1073741824 [ 3.145845] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.150276] vdc: detected capacity change from 0 to 2621440000 [ 3.166830] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.171442] vdd: detected capacity change from 0 to 2621440000 [ 3.188669] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.191882] vde: detected capacity change from 0 to 4294967296 [ 3.207509] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.210837] vdf: detected capacity change from 0 to 4294967296 [ 3.218983] libphy: Fixed MDIO Bus: probed [ 3.223697] usbcore: registered new interface driver usbserial_generic [ 3.226868] usbserial: USB Serial support registered for generic [ 3.230716] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.235760] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.237979] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.240634] mousedev: PS/2 mouse device common for all mice [ 3.243401] rtc_cmos 00:05: RTC can wake from S4 [ 3.245758] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.249187] rtc_cmos 00:05: registered as rtc0 [ 3.251501] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.252565] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.254305] intel_pstate: CPU model not supported [ 3.259208] hid: raw HID events driver (C) Jiri Kosina [ 3.259827] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.263088] usbcore: registered new interface driver usbhid [ 3.267980] usbhid: USB HID core driver [ 3.269829] drop_monitor: Initializing network drop monitor service [ 3.272870] Initializing XFRM netlink socket [ 3.275291] NET: Registered protocol family 10 [ 3.278968] Segment Routing with IPv6 [ 3.280610] NET: Registered protocol family 17 [ 3.283000] mpls_gso: MPLS GSO support [ 3.289248] RAS: Correctable Errors collector initialized. [ 3.291968] AVX version of gcm_enc/dec engaged. [ 3.294026] AES CTR mode by8 optimization enabled [ 3.369720] sched_clock: Marking stable (3369687033, 0)->(4300076878, -930389845) [ 3.373902] registered taskstats version 1 [ 3.376092] Loading compiled-in X.509 certificates [ 3.377971] zswap: loaded using pool lzo/zbud [ 3.427818] Key type big_key registered [ 3.443662] Key type encrypted registered [ 3.445669] ima: No TPM chip found, activating TPM-bypass! [ 3.448117] ima: Allocated hash algorithm: sha1 [ 3.450142] ima: No architecture policies found [ 3.452213] evm: Initialising EVM extended attributes: [ 3.454445] evm: security.selinux [ 3.455938] evm: security.ima [ 3.457187] evm: security.capability [ 3.458792] evm: HMAC attrs: 0x1 [ 3.462459] rtc_cmos 00:05: setting system clock to 2026-09-19 01:25:11 UTC (1789781111) [ 3.468149] debug: unmapping init [mem 0xffffffff86003000-0xffffffff861fffff] [ 3.470818] debug: unmapping init [mem 0xffffffff84d82000-0xffffffff85058fff] [ 3.479141] Write protecting the kernel read-only data: 28672k [ 3.483100] debug: unmapping init [mem 0xffffffff83403000-0xffffffff835fffff] [ 3.486272] debug: unmapping init [mem 0xffffffff83d14000-0xffffffff83dfffff] [ 3.529550] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.540227] systemd[1]: Detected virtualization kvm. [ 3.542510] systemd[1]: Detected architecture x86-64. [ 3.544244] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.571950] systemd[1]: No hostname configured. [ 3.573910] systemd[1]: Set hostname to . [ 3.576284] random: systemd: uninitialized urandom read (16 bytes read) [ 3.580496] systemd[1]: Initializing machine ID from random generator. [ 3.626939] random: ln: uninitialized urandom read (6 bytes read) [ 3.736246] random: systemd: uninitialized urandom read (16 bytes read) [ 3.739202] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.744264] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.749633] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Swap. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Timers. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Journal Service. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.447290] device-mapper: uevent: version 1.0.3 [ 4.449952] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.112599] random: fast init done [ 5.124207] virtio_net virtio0 ens2: renamed from eth0 [ 5.157984] scsi host0: ata_piix [ 5.274152] scsi host1: ata_piix [ 5.276095] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.279831] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.979459] dracut-initqueue[580]: RTNETLINK answers: File exists [ 9.965548] random: crng init done [ 9.968681] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.462872] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.643817] printk: systemd: 23 output lines suppressed due to ratelimiting [ 11.894110] SELinux: Disabled at runtime. [ 11.956915] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.966546] systemd[1]: Detected virtualization kvm. [ 11.968547] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.418394] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.420956] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.424952] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.428710] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.431157] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.438613] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.445031] systemd[1]: Activating swap /dev/disk/by-label/SWAP... Activating swap /dev/disk/by-label/SWAP... Mounting POSIX Message Queue File System... Starting Remount Root and Kernel File Systems... [ OK ] Created slice User and Session Slice. [ 12.471703] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Control Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Starting Apply Kernel Variables... Mounting Huge Pages File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Mounting Kernel Debug File System... [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target Slices. Starting udev Coldplug all Devices... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.836955] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.066474] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.106348] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.206138] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.217304] EDAC sbridge: Ver: 1.1.2 [ 15.052887] Key type dns_resolver registered [ 15.353555] NFS: Registering the id_resolver key type [ 15.356350] Key type id_resolver registered [ 15.358028] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg345-server login: [ 37.088014] hrtimer: interrupt took 18318749 ns [ 74.528137] libcfs: loading out-of-tree module taints kernel. [ 74.590819] Key type ._llcrypt registered [ 74.593996] Key type .llcrypt registered [ 74.707364] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 92.689745] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 94.220546] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 94.236517] alg: No test for adler32 (adler32-zlib) [ 95.649144] Lustre: Lustre: Build Version: 2.17.57_110_g5f5d390 [ 96.482040] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 98.215276] Key type lgssc registered [ 99.999240] Lustre: Echo OBD driver; http://www.lustre.org/ [ 117.132198] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 158.498731] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 171.339287] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 171.367338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 172.670327] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 172.709030] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 172.810943] Lustre: lustre-MDT0000: new disk, initializing [ 172.915612] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 172.937965] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 177.036808] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 192.287146] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 192.418438] Lustre: 6503:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 192.450175] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 192.453131] Lustre: Skipped 1 previous similar message [ 192.573320] Lustre: lustre-MDT0001: new disk, initializing [ 192.659432] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 192.703045] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 192.712362] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 198.193365] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 203.645211] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 214.027593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 214.324589] Lustre: lustre-OST0000: new disk, initializing [ 214.327533] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 214.333562] Lustre: 8442:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 214.427912] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 219.649766] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 219.655337] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 219.731858] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 221.259273] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 236.787162] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 236.924276] Lustre: lustre-OST0001: new disk, initializing [ 236.931450] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 236.941449] Lustre: 9514:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 237.041343] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 243.722266] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 246.841862] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 246.855196] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 246.904564] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 256.210679] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 262.404434] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 269.272872] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing check_logdir /tmp/testlogs/ [ 274.598973] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing yml_node [ 279.415449] Lustre: DEBUG MARKER: Client: 2.17.57.110 [ 282.353216] Lustre: DEBUG MARKER: MDS: 2.17.57.110 [ 285.570606] Lustre: DEBUG MARKER: OSS: 2.17.57.110 [ 287.485633] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Sep 18 21:29:54 EDT 2026 [ 306.398550] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 308.105373] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 326.114490] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 326.124529] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 326.153787] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 328.673273] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 328.674425] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 328.687023] Lustre: Skipped 1 previous similar message [ 328.697916] Lustre: Skipped 1 previous similar message [ 330.185078] Lustre: server umount lustre-MDT0000 complete [ 333.998056] LustreError: 6496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789781442 with bad export cookie 17969346981660657774 [ 334.010757] LustreError: 6496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 334.011140] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 338.912259] LustreError: 6512:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 338.926558] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 338.927447] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 338.936303] LustreError: 6512:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 338.953848] Lustre: Skipped 2 previous similar messages [ 339.945474] LustreError: 6510:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 339.962959] LustreError: 6510:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 340.380488] Lustre: server umount lustre-MDT0001 complete [ 345.151364] Lustre: server umount lustre-OST0000 complete [ 349.477841] Lustre: server umount lustre-OST0001 complete [ 356.565556] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 367.314541] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 405.693630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 406.106873] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 406.187072] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 406.335435] Lustre: lustre-MDT0000: new disk, initializing [ 406.419421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 406.434947] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 411.640241] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 423.549324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 423.748923] Lustre: 15069:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 423.831973] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 423.840119] Lustre: Skipped 1 previous similar message [ 423.979739] Lustre: lustre-MDT0001: new disk, initializing [ 424.184638] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 424.225956] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 424.246404] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 429.626848] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 440.056715] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 446.165829] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 455.161875] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 455.766744] Lustre: lustre-OST0000: new disk, initializing [ 455.769541] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 455.788198] Lustre: 17040:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 455.963334] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 457.498214] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 457.519913] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 457.626406] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 464.276512] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 476.373748] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 481.474674] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 481.722983] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 486.633982] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 486.943164] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 493.555983] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 493.560460] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 493.570827] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 493.570836] Lustre: Skipped 3 previous similar messages [ 493.583817] Lustre: Skipped 1 previous similar message [ 493.623934] LustreError: Skipped 1 previous similar message [ 498.657040] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 498.664265] Lustre: Skipped 2 previous similar messages [ 503.263109] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 503.466710] Lustre: server umount lustre-OST0000 complete [ 508.898198] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 508.900682] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 508.918551] Lustre: Skipped 1 previous similar message [ 508.942269] Lustre: Skipped 1 previous similar message [ 513.513188] Lustre: server umount lustre-MDT0000 complete [ 514.023554] LustreError: 15081:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 514.041899] LustreError: 15081:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 516.863820] LustreError: 15866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789781624 with bad export cookie 17969346981660659307 [ 516.875992] LustreError: 15866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 516.877951] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 517.173632] Lustre: server umount lustre-MDT0001 complete [ 521.528414] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 21:33:48 (1789781628) [ 524.197974] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 526.317439] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 21:33:53 (1789781633) [ 533.192089] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 533.693200] LustreError: 19450:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 533.793718] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 538.083680] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 539.110888] LustreError: 19451:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 546.403493] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 550.321968] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 558.674295] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 564.439389] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 572.234719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 579.522747] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 587.243915] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 599.528213] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 599.544519] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 599.577339] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 601.577292] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 604.027173] Lustre: server umount lustre-OST0000 complete [ 615.107951] Lustre: server umount lustre-MDT0000 complete [ 616.932733] LustreError: 19451:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 616.950844] LustreError: 19451:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 618.946190] LustreError: 19430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789781726 with bad export cookie 17969346981660660189 [ 618.955571] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 618.962894] LustreError: 19430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 619.216614] Lustre: server umount lustre-MDT0001 complete [ 630.323881] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 633.540340] Key type lgssc unregistered [ 633.825206] LNet: 22983:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 633.829647] LNetError: 22983:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 633.851205] LNet: Removed LNI 192.168.203.145@tcp [ 634.831151] Key type .llcrypt unregistered [ 634.833627] Key type ._llcrypt unregistered [ 656.073771] Key type ._llcrypt registered [ 656.078074] Key type .llcrypt registered [ 656.189704] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 657.429272] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 657.455257] alg: No test for adler32 (adler32-zlib) [ 658.574364] Lustre: Lustre: Build Version: 2.17.57_110_g5f5d390 [ 658.793859] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 660.447185] Key type lgssc registered [ 661.694830] Lustre: Echo OBD driver; http://www.lustre.org/ [ 669.438310] loop: module loaded [ 669.513398] loop0: detected capacity change from 0 to 204800000 [ 669.756209] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 669.773608] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 670.072087] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 676.211512] loop0: detected capacity change from 0 to 204800000 [ 676.377740] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 676.387039] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 676.395982] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 681.406814] loop0: detected capacity change from 0 to 204800000 [ 686.721035] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 686.739275] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 687.954875] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 688.039682] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 688.186831] Lustre: lustre-MDT0000: new disk, initializing [ 688.283450] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 688.322853] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 693.285808] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 703.243647] loop1: detected capacity change from 0 to 204800000 [ 709.199922] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 709.434729] Lustre: lustre-OST07c6: new disk, initializing [ 709.438954] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 709.446295] Lustre: Skipped 1 previous similar message [ 709.450878] Lustre: 26165:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 709.520112] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 715.378525] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 715.396935] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 715.445730] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 715.447777] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 719.990447] Lustre: 26680:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 255 < left 264, rollback = 2 [ 719.996951] Lustre: 26680:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 720.001421] Lustre: 26680:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 720.007494] Lustre: 26680:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 720.015183] Lustre: 26680:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 720.022252] Lustre: 26680:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 740.825665] Lustre: 25153:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 740.843249] Lustre: 25153:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 740.856290] Lustre: 25153:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 740.861554] Lustre: 25153:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 740.868153] Lustre: 25153:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 740.873633] Lustre: 25153:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 746.173179] Lustre: 25153:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 746.182988] Lustre: 25153:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 746.196285] Lustre: 25153:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 746.207627] Lustre: 25153:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 746.220604] Lustre: 25153:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 746.226409] Lustre: 25153:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 746.236887] Lustre: 25153:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 746.251709] Lustre: 25153:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 746.254853] Lustre: 25153:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 746.270391] Lustre: 25153:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 746.274501] Lustre: 25153:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 746.279414] Lustre: 25153:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 749.507649] Lustre: 25151:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 749.522605] Lustre: 25151:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 749.537876] Lustre: 25151:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 749.551836] Lustre: 25151:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 749.561985] Lustre: 25151:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 749.573634] Lustre: 25151:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 749.579924] Lustre: 25151:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 749.591680] Lustre: 25151:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 749.608345] Lustre: 25151:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 749.614574] Lustre: 25151:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 749.623048] Lustre: 25151:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 749.635927] Lustre: 25151:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 754.724721] Lustre: 25152:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 754.738520] Lustre: 25152:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 754.753195] Lustre: 25152:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 754.758399] Lustre: 25152:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 754.764380] Lustre: 25152:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 754.769848] Lustre: 25152:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 754.775621] Lustre: 25152:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 754.787917] Lustre: 25152:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 754.793820] Lustre: 25152:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 754.800481] Lustre: 25152:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 754.808202] Lustre: 25152:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 754.814587] Lustre: 25152:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 765.710712] Lustre: 25153:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 765.721456] Lustre: 25153:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 765.738254] Lustre: 25153:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 765.748913] Lustre: 25153:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 765.758580] Lustre: 25153:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 765.773517] Lustre: 25153:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 765.782606] Lustre: 25153:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 765.791780] Lustre: 25153:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 765.795705] Lustre: 25153:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 765.800759] Lustre: 25153:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 765.815495] Lustre: 25153:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 765.825901] Lustre: 25153:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 817.640209] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 817.640521] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 817.666569] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 820.739931] Lustre: server umount lustre-OST07c6 complete [ 832.129749] Lustre: server umount lustre-MDT0000 complete [ 849.684296] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 853.059467] Key type lgssc unregistered [ 853.493780] LNet: 28748:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 853.502754] LNetError: 28748:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 854.569097] LNet: Removed LNI 192.168.203.145@tcp [ 855.510495] Key type .llcrypt unregistered [ 855.513234] Key type ._llcrypt unregistered [ 874.683635] Key type ._llcrypt registered [ 874.687281] Key type .llcrypt registered [ 874.832897] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 875.863514] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 875.895995] alg: No test for adler32 (adler32-zlib) [ 876.900061] Lustre: Lustre: Build Version: 2.17.57_110_g5f5d390 [ 877.149453] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 878.855220] Key type lgssc registered [ 880.084814] Lustre: Echo OBD driver; http://www.lustre.org/ [ 888.204433] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 888.227422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 889.649117] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 894.291585] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 899.043442] LustreError: 30015:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 899.055958] LustreError: 30015:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 902.790292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 903.080888] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 906.833051] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 914.542789] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 921.829313] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 929.142512] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 929.569711] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 934.704157] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 935.698416] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 943.999363] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 968.672423] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 968.677712] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 968.685305] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 968.693468] Lustre: Skipped 1 previous similar message [ 968.707862] Lustre: Skipped 1 previous similar message [ 972.062926] Lustre: server umount lustre-OST0000 complete [ 978.915157] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 978.925659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 978.932408] Lustre: Skipped 1 previous similar message [ 978.951166] Lustre: Skipped 1 previous similar message [ 984.036149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 984.039968] Lustre: Skipped 1 previous similar message [ 988.129731] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 988.135709] Lustre: Skipped 1 previous similar message [ 991.199127] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 991.361218] Lustre: server umount lustre-MDT0000 complete [ 993.248390] LustreError: 30010:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 993.271086] LustreError: 30010:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 995.577194] LustreError: 31861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789782103 with bad export cookie 7399457994663999244 [ 995.594305] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 995.595751] LustreError: 31861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 995.977340] Lustre: server umount lustre-MDT0001 complete [ 1006.564790] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 1009.745182] Key type lgssc unregistered [ 1010.028799] LNet: 33756:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1010.034703] LNetError: 33756:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1010.045725] LNet: Removed LNI 192.168.203.145@tcp [ 1011.005307] Key type .llcrypt unregistered [ 1011.009176] Key type ._llcrypt unregistered [ 1031.722315] Key type ._llcrypt registered [ 1031.730555] Key type .llcrypt registered [ 1031.885895] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 1033.089344] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1033.188857] alg: No test for adler32 (adler32-zlib) [ 1034.391221] Lustre: Lustre: Build Version: 2.17.57_110_g5f5d390 [ 1034.683450] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 1036.383172] Key type lgssc registered [ 1037.623455] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1045.761846] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1045.789467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1047.613549] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1052.925280] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1057.766335] LustreError: 35022:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1057.799765] LustreError: 35022:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1062.212531] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1062.648706] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1067.948362] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1075.809428] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1081.582856] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1089.510563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1090.014220] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1092.096482] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1098.959955] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1108.641667] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1122.786161] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1122.805139] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1122.815311] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1125.007409] Lustre: server umount lustre-OST0000 complete [ 1129.441686] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1129.455681] LustreError: Skipped 1 previous similar message [ 1129.463726] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1129.480286] Lustre: Skipped 1 previous similar message [ 1129.490705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1129.495867] Lustre: Skipped 1 previous similar message [ 1133.025886] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1133.026577] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1133.065510] Lustre: Skipped 1 previous similar message [ 1134.970177] Lustre: server umount lustre-MDT0000 complete [ 1138.151973] LustreError: 35022:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1138.180060] LustreError: 35022:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1138.667528] LustreError: 35004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789782246 with bad export cookie 10802384355828692524 [ 1138.689466] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1139.178956] Lustre: server umount lustre-MDT0001 complete [ 1149.659245] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 1152.268688] Key type lgssc unregistered [ 1152.510672] LNet: 38556:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1152.519214] LNetError: 38556:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1152.530546] LNet: Removed LNI 192.168.203.145@tcp [ 1153.471230] Key type .llcrypt unregistered [ 1153.472953] Key type ._llcrypt unregistered [ 1177.437481] Key type ._llcrypt registered [ 1177.439985] Key type .llcrypt registered [ 1177.555596] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 1178.949225] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1178.973487] alg: No test for adler32 (adler32-zlib) [ 1180.084434] Lustre: Lustre: Build Version: 2.17.57_110_g5f5d390 [ 1180.425659] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 1182.119890] Key type lgssc registered [ 1183.554502] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1191.992602] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1192.037847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1193.593780] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1198.353932] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1203.181214] LustreError: 39969:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1203.203776] LustreError: 39969:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1207.165415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1207.584944] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1212.821809] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1221.148968] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1228.237602] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1235.401730] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1235.827223] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1238.908533] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1242.212602] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1250.272673] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1257.462881] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1257.467758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1257.484764] Lustre: Skipped 1 previous similar message [ 1257.509910] Lustre: Skipped 2 previous similar messages [ 1257.809770] Lustre: server umount lustre-MDT0000 complete [ 1258.466096] LustreError: 39964:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1262.035411] LustreError: 39952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789782370 with bad export cookie 10603147527340381290 [ 1262.055706] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1262.058179] LustreError: 39952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1262.561165] LustreError: 41270:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1262.562338] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1262.596359] Lustre: Skipped 1 previous similar message [ 1262.599902] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1267.680644] LustreError: 39966:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1267.681061] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1267.729595] LustreError: 39966:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1267.737407] Lustre: Skipped 1 previous similar message [ 1272.802200] LustreError: 39966:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1272.802966] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1272.818881] LustreError: 39966:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1272.835347] Lustre: Skipped 1 previous similar message [ 1276.383289] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1276.622209] Lustre: server umount lustre-MDT0001 complete [ 1297.770957] Lustre: server umount lustre-OST0000 complete [ 1317.962463] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 1320.966820] Key type lgssc unregistered [ 1321.169064] LNet: 43831:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1321.204683] LNetError: 43831:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1321.226978] LNet: Removed LNI 192.168.203.145@tcp [ 1321.809301] Key type .llcrypt unregistered [ 1321.814702] Key type ._llcrypt unregistered [ 1352.094572] Key type ._llcrypt registered [ 1352.096588] Key type .llcrypt registered [ 1352.165691] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 1353.373791] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1353.435612] alg: No test for adler32 (adler32-zlib) [ 1354.606765] Lustre: Lustre: Build Version: 2.17.57_110_g5f5d390 [ 1354.870987] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 1356.559471] Key type lgssc registered [ 1357.759669] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1366.316753] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1366.336563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1367.727626] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1372.795465] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1377.255937] LustreError: 45246:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1377.282867] LustreError: 45246:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1382.370356] LustreError: 45247:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1383.143294] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1383.585229] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1388.432561] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1395.883584] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1403.774822] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1413.096535] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1413.746946] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1421.828740] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1430.370211] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1437.307860] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1443.808653] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1443.819911] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1443.853100] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1447.885619] Lustre: lustre-OST0000: Not available for connect from 192.168.203.45@tcp (stopping) [ 1447.904966] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1447.913032] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1449.427795] Lustre: server umount lustre-OST0000 complete [ 1468.384729] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1468.387645] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1468.411457] Lustre: Skipped 1 previous similar message [ 1468.436946] Lustre: Skipped 3 previous similar messages [ 1473.504991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1473.519586] Lustre: Skipped 1 previous similar message [ 1473.748324] Lustre: server umount lustre-MDT0000 complete [ 1474.530347] LustreError: 45246:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1474.558126] LustreError: 45246:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1477.600462] LustreError: 45229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789782585 with bad export cookie 13095926432608648303 [ 1477.613072] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1477.617196] LustreError: 45229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1484.000571] Lustre: server umount lustre-MDT0001 complete [ 1496.384573] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 1498.946522] Key type lgssc unregistered [ 1499.226583] LNet: 48974:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1499.231742] LNetError: 48974:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1499.248685] LNet: Removed LNI 192.168.203.145@tcp [ 1500.034428] Key type .llcrypt unregistered [ 1500.038490] Key type ._llcrypt unregistered [ 1530.050221] Key type ._llcrypt registered [ 1530.052825] Key type .llcrypt registered [ 1530.173282] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 1531.225192] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1531.279754] alg: No test for adler32 (adler32-zlib) [ 1532.349774] Lustre: Lustre: Build Version: 2.17.57_110_g5f5d390 [ 1532.642951] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 1534.313329] Key type lgssc registered [ 1535.651920] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1545.249235] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1545.290288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1546.937788] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1552.077526] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1556.450572] LustreError: 50389:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1556.480387] LustreError: 50389:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1561.515114] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1561.571731] LustreError: 50388:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1561.985725] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1565.958314] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1574.151508] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1581.270646] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1591.152548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1591.543193] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1592.573351] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1598.400156] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1607.823115] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1614.549912] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1618.554633] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1630.991966] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-1821h52m17s [ 1632.793632] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1636.295886] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.45@tcp (stopping) [ 1637.345626] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.45@tcp (stopping) [ 1637.859568] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1637.884933] Lustre: Skipped 2 previous similar messages [ 1638.949115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1638.975893] Lustre: Skipped 5 previous similar messages [ 1641.162413] Lustre: server umount lustre-MDT0000 complete [ 1642.399555] LustreError: 50385:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1644.813967] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1645.028719] LustreError: 50389:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1645.050397] LustreError: 50389:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1651.174224] LustreError: 50388:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1651.196401] LustreError: 50388:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1652.448541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1652.523148] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1652.727095] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1652.792689] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1653.740929] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1657.885325] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1661.798612] Lustre: DEBUG MARKER: Wait for df (36679) ... [ 1663.285960] Lustre: DEBUG MARKER: done [ 1668.064394] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1668.069222] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1668.077482] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1668.579418] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1674.218185] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1674.235682] Lustre: Skipped 4 previous similar messages [ 1681.375142] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1681.545892] Lustre: server umount lustre-OST0000 complete [ 1689.574851] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1689.577040] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1689.595086] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1689.615965] Lustre: Skipped 2 previous similar messages [ 1689.624717] Lustre: Skipped 2 previous similar messages [ 1691.662044] Lustre: server umount lustre-MDT0000 complete [ 1695.596949] LustreError: 50370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789782803 with bad export cookie 15170749209936343597 [ 1695.601255] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1695.623468] LustreError: 50370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1696.053613] Lustre: server umount lustre-MDT0001 complete [ 1707.944318] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 1711.294559] Key type lgssc unregistered [ 1711.617078] LNet: 55317:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1711.641801] LNetError: 55317:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1711.655958] LNet: Removed LNI 192.168.203.145@tcp [ 1712.432467] Key type .llcrypt unregistered [ 1712.435705] Key type ._llcrypt unregistered [ 1758.007784] Key type ._llcrypt registered [ 1758.009808] Key type .llcrypt registered [ 1758.121350] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 1759.468092] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1759.556693] alg: No test for adler32 (adler32-zlib) [ 1760.766955] Lustre: Lustre: Build Version: 2.17.57_110_g5f5d390 [ 1761.055992] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 1762.775121] Key type lgssc registered [ 1764.072055] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1772.716337] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1772.735943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1774.014523] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1774.026328] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1774.164412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1778.704464] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1789.186570] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1789.384164] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1789.410715] Lustre: 57525:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1789.759765] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1795.654491] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1804.530238] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1810.364430] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1817.040145] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1817.284052] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1817.474810] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1825.735469] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1828.101783] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1835.242253] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1842.360594] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-1821h55m49s [ 1843.784973] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1850.821667] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1853.373596] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1853.428498] LustreError: 57531:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1873.906252] Lustre: lustre-MDT0000: Client a6c9d877-1089-43c3-bbe2-2d9c59e51c03 (at 192.168.203.45@tcp) reconnecting [ 1893.455718] LustreError: 57531:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1895.886938] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1898.499681] Lustre: DEBUG MARKER: done [ 1904.578965] Lustre: server umount lustre-OST0000 complete [ 1910.251954] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1910.261249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1910.272301] Lustre: Skipped 1 previous similar message [ 1911.331408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1911.337412] Lustre: Skipped 1 previous similar message [ 1917.422329] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1917.431474] Lustre: Skipped 1 previous similar message [ 1918.501307] Lustre: server umount lustre-MDT0000 complete [ 1921.327404] LustreError: 58277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789783029 with bad export cookie 14962464977005599720 [ 1921.334678] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1921.337645] LustreError: 58277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1931.930106] Lustre: server umount lustre-MDT0001 complete [ 1943.900928] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 1947.579169] Key type lgssc unregistered [ 1947.802066] LNet: 61858:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1947.820627] LNetError: 61858:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1947.832833] LNet: Removed LNI 192.168.203.145@tcp [ 1948.673408] Key type .llcrypt unregistered [ 1948.675391] Key type ._llcrypt unregistered [ 1996.021755] Key type ._llcrypt registered [ 1996.024206] Key type .llcrypt registered [ 1996.164744] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 1997.782354] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1997.873269] alg: No test for adler32 (adler32-zlib) [ 1999.154884] Lustre: Lustre: Build Version: 2.17.57_110_g5f5d390 [ 1999.529464] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 2001.264587] Key type lgssc registered [ 2002.658909] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2007.672586] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2007.690446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2009.091601] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2009.141265] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2009.475980] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2010.824161] Lustre: Failing over lustre-MDT0000 [ 2011.551724] Lustre: server umount lustre-MDT0000 complete [ 2019.861109] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 21:58:47 (1789783127) [ 2026.299484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2026.815603] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2030.942093] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2038.629880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2038.739634] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2038.786443] Lustre: 64660:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2039.003871] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2043.632075] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2051.461593] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2057.371889] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2064.610279] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2064.818392] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2065.007263] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2070.464212] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2074.291228] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2077.658794] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2083.132953] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2089.956693] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2089.963085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2089.983410] Lustre: Skipped 2 previous similar messages [ 2090.002418] Lustre: Skipped 2 previous similar messages [ 2095.072509] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2095.086289] Lustre: Skipped 2 previous similar messages [ 2095.606028] Lustre: server umount lustre-MDT0000 complete [ 2100.042461] LustreError: 64651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789783208 with bad export cookie 17061997069835750504 [ 2100.048222] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2100.088930] LustreError: 64651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2100.192512] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2100.207184] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2100.468021] Lustre: server umount lustre-MDT0001 complete [ 2104.623964] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2109.408694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2116.992160] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2125.794831] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xecc863efca50d54b [ 2125.802869] Lustre: MGC192.168.203.145@tcp: Connection restored to 0@lo (at 0@lo) [ 2126.117561] LustreError: 66509:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2126.134975] LustreError: 66509:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2126.220666] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2126.306978] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2130.339068] LustreError: 66510:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2130.356542] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2131.174452] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2135.525535] LustreError: 66509:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2139.360063] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2139.765561] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2140.791662] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2144.423027] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2153.271496] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2160.096071] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2170.342301] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2170.349567] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2170.358661] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2173.711346] Lustre: server umount lustre-MDT0000 complete [ 2176.992920] LustreError: 69227:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2177.016888] LustreError: 69227:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2177.562120] LustreError: 65396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789783285 with bad export cookie 17061997069835752779 [ 2177.576985] LustreError: 65396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2177.577623] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2177.986472] Lustre: server umount lustre-MDT0001 complete [ 2185.215489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2196.221608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2198.497039] Lustre: 63579:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789783290/real 1789783290] req@000000008090caa8 x1876723611274240/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789783306 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2198.527399] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2198.537843] Lustre: Skipped 2 previous similar messages [ 2203.043523] LustreError: 66509:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2203.154374] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2203.242236] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2203.679291] Lustre: 63580:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789783295/real 1789783295] req@00000000479097ee x1876723611274368/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789783311 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2208.342393] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2209.183728] Lustre: 63582:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789783300/real 1789783300] req@000000001bf4264f x1876723611274624/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789783316 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2213.344283] Lustre: 63582:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789783305/real 1789783305] req@0000000020a75698 x1876723611274752/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789783321 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2213.347595] LustreError: 66510:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2213.405977] LustreError: 66510:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2217.504783] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2223.099427] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2223.403516] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2227.247865] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2232.699653] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2239.410376] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2247.658944] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2247.667380] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2247.679298] Lustre: Skipped 1 previous similar message [ 2247.687841] Lustre: Skipped 3 previous similar messages [ 2252.904215] Lustre: server umount lustre-MDT0000 complete [ 2257.074308] LustreError: 64651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789783365 with bad export cookie 17061997069835754158 [ 2257.075879] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2257.083297] LustreError: 64651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2257.416411] Lustre: server umount lustre-MDT0001 complete [ 2264.098372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2267.634193] Lustre: DEBUG MARKER: files compared the same [ 2279.987697] Lustre: server umount lustre-OST0000 complete [ 2298.089300] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 2300.985178] Key type lgssc unregistered [ 2301.300347] LNet: 74400:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2301.310566] LNetError: 74400:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2301.334203] LNet: Removed LNI 192.168.203.145@tcp [ 2302.132382] Key type .llcrypt unregistered [ 2302.134405] Key type ._llcrypt unregistered [ 2327.889869] Key type ._llcrypt registered [ 2327.892844] Key type .llcrypt registered [ 2328.044944] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 2329.173889] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2329.337478] alg: No test for adler32 (adler32-zlib) [ 2330.511611] Lustre: Lustre: Build Version: 2.17.57_110_g5f5d390 [ 2330.739517] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 2332.399270] Key type lgssc registered [ 2333.863774] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2343.299837] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2343.336144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2344.913802] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2350.186687] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug +malloc all [ 2354.150370] LustreError: 75816:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2354.162513] LustreError: 75816:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2358.849997] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2359.161064] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2363.873632] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug +malloc all [ 2372.019835] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2378.948954] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2385.124830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2385.376990] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2389.490094] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2391.555822] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug +malloc all [ 2401.438345] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2415.072160] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2415.079058] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2415.093584] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2415.103289] Lustre: Skipped 1 previous similar message [ 2420.192255] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2420.196231] Lustre: Skipped 1 previous similar message [ 2421.259461] Lustre: server umount lustre-OST0000 complete [ 2425.313066] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2425.317691] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2425.326663] Lustre: Skipped 2 previous similar messages [ 2425.346802] Lustre: Skipped 1 previous similar message [ 2430.432950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2430.436354] Lustre: Skipped 1 previous similar message [ 2431.349284] Lustre: server umount lustre-MDT0000 complete [ 2434.934220] LustreError: 76552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789783542 with bad export cookie 13593894146447908685 [ 2434.938847] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2434.945436] LustreError: 76552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2435.261448] Lustre: server umount lustre-MDT0001 complete [ 2446.461762] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 2449.178457] Key type lgssc unregistered [ 2449.586511] LNet: 79371:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2449.597883] LNetError: 79371:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2449.636559] LNet: Removed LNI 192.168.203.145@tcp [ 2450.728081] Key type .llcrypt unregistered [ 2450.730422] Key type ._llcrypt unregistered [ 2478.730913] Key type ._llcrypt registered [ 2478.732643] Key type .llcrypt registered [ 2478.849685] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 2480.020890] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2480.083203] alg: No test for adler32 (adler32-zlib) [ 2481.109873] Lustre: Lustre: Build Version: 2.17.57_110_g5f5d390 [ 2481.362139] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 2483.096236] Key type lgssc registered [ 2484.475444] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2494.421316] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2494.458407] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2510.111384] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2515.231325] LustreError: 80759:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.145@tcp: failed processing log, type 4: rc = -110 [ 2541.921522] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2548.062562] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2554.923295] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2562.408138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2562.547330] Lustre: *** cfs_fail_loc=706, val=0*** [ 2562.559445] LustreError: 80092:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000003ff43545 x1876724116621568/t0(0) o250->MGC192.168.203.145@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2562.912374] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2562.999566] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2567.622805] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2568.168578] LustreError: 80784:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2568.189856] LustreError: 80784:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2573.283068] LustreError: 80785:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2575.943174] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2576.368897] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2580.687624] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2588.232988] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2594.387186] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2596.835294] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2596.841509] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2596.852065] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2597.856623] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2597.870856] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2602.926944] Lustre: server umount lustre-OST0000 complete [ 2612.198276] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2612.221149] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2612.270138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2612.284063] Lustre: Skipped 1 previous similar message [ 2614.939760] Lustre: server umount lustre-MDT0000 complete [ 2618.336509] LustreError: 82926:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2618.359056] LustreError: 82926:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2620.573839] LustreError: 83625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789783728 with bad export cookie 16091768742898052684 [ 2620.585943] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2620.596132] LustreError: 83625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2620.988593] Lustre: server umount lustre-MDT0001 complete [ 2634.520898] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 2638.487746] Key type lgssc unregistered [ 2638.935388] LNet: 84377:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2638.952666] LNetError: 84377:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2638.977506] LNet: Removed LNI 192.168.203.145@tcp [ 2640.227341] Key type .llcrypt unregistered [ 2640.229853] Key type ._llcrypt unregistered [ 2669.402735] Key type ._llcrypt registered [ 2669.407377] Key type .llcrypt registered [ 2669.487309] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 2670.424120] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2670.465103] alg: No test for adler32 (adler32-zlib) [ 2671.435141] Lustre: Lustre: Build Version: 2.17.57_110_g5f5d390 [ 2671.647689] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 2673.352474] Key type lgssc registered [ 2674.918721] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2683.490290] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2683.528158] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2689.072667] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2696.866833] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2697.466883] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2701.872544] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2702.818889] LustreError: 86476:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2702.839231] LustreError: 86476:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2707.936828] LustreError: 86480:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2707.956922] LustreError: 86480:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2710.848605] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2711.314340] LustreError: 86479:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2711.357777] LustreError: 86479:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2711.380788] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2716.643425] LustreError: 86480:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2716.671044] LustreError: 86480:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2718.244140] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2721.763823] LustreError: 86479:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2721.786905] LustreError: 86479:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2727.355724] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2727.366547] Lustre: MGS: Client 2e5055cc-68c5-4afc-8ca4-a7dbf62a9b58 (at 0@lo) reconnecting [ 2727.392531] LustreError: 86490:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2727.603702] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2727.714302] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2732.124627] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2748.388605] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2748.401168] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2748.401454] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2748.402327] Lustre: Skipped 1 previous similar message [ 2748.416224] Lustre: Skipped 1 previous similar message [ 2751.344730] Lustre: server umount lustre-OST0000 complete [ 2758.624628] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2758.625825] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2758.651861] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2758.684043] Lustre: Skipped 1 previous similar message [ 2758.699785] Lustre: Skipped 1 previous similar message [ 2762.519901] Lustre: server umount lustre-MDT0000 complete [ 2763.752727] LustreError: 86475:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2763.796516] LustreError: 86475:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2768.447596] Lustre: server umount lustre-MDT0001 complete [ 2773.279636] Lustre: server umount MGS complete [ 2786.819896] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 22:11:33 (1789783893) [ 2820.096778] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 2829.967105] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 2863.897698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2868.781325] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2879.640575] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2879.838510] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2879.875919] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2879.950404] Lustre: lustre-MDT0001: new disk, initializing [ 2880.090606] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2880.180845] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2880.190730] Lustre: 93650:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2880.205338] LustreError: 93650:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2885.024471] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2896.315464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2896.662379] Lustre: lustre-OST0000: new disk, initializing [ 2896.666524] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2896.673194] Lustre: 94416:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2896.744373] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2902.492933] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2907.238885] Lustre: 94889:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000829e1640 x1876724315889408/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1789784031 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2913.767156] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2921.614102] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2921.642326] Lustre: MGS: Client 452e0a98-b27a-4989-8d3b-0e351ce13fdb (at 0@lo) reconnecting [ 2921.671823] LustreError: 93636:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2921.710665] Lustre: 95462:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2921.771224] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2921.929586] Lustre: lustre-MDT0000: new disk, initializing [ 2922.162978] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2922.222925] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2922.264288] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2923.303452] Lustre: 95486:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000002678bb04 x1876724315896576/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1789784047 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2927.530870] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2928.486872] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2932.463233] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2932.473616] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2932.569800] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2960.352452] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2960.359449] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2960.369195] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2965.361790] Lustre: server umount lustre-OST0000 complete [ 2971.618567] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2971.621434] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2971.625785] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2971.625796] Lustre: Skipped 2 previous similar messages [ 2971.646756] Lustre: Skipped 2 previous similar messages [ 2975.561150] Lustre: server umount lustre-MDT0000 complete [ 2976.736964] LustreError: 95733:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2976.758562] LustreError: 95733:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2985.587590] Lustre: server umount lustre-MDT0001 complete [ 2989.652557] Lustre: server umount MGS complete [ 3001.103801] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 22:15:08 (1789784108) [ 3021.755349] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 3025.329159] Key type lgssc unregistered [ 3025.745663] LNet: 97985:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3025.766270] LNetError: 97985:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3025.809100] LNet: Removed LNI 192.168.203.145@tcp [ 3027.216465] Key type .llcrypt unregistered [ 3027.219240] Key type ._llcrypt unregistered [ 3047.382333] Key type ._llcrypt registered [ 3047.387908] Key type .llcrypt registered [ 3047.529981] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 3048.830684] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3048.890574] alg: No test for adler32 (adler32-zlib) [ 3049.996336] Lustre: Lustre: Build Version: 2.17.57_110_g5f5d390 [ 3050.283234] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 3052.007273] Key type lgssc registered [ 3053.301216] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3061.052520] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3061.075313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3062.693777] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3068.059616] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3071.976615] LustreError: 99209:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3071.996430] LustreError: 99209:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3077.097694] LustreError: 99210:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3077.267586] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3077.505939] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3082.979741] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3090.702657] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3096.485830] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3103.751220] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3104.156503] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3107.252671] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3110.638897] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3117.924274] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3132.898771] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3132.899081] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3132.900380] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3132.921579] Lustre: Skipped 1 previous similar message [ 3134.808901] Lustre: server umount lustre-OST0000 complete [ 3143.139643] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3143.153032] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3143.165410] Lustre: Skipped 2 previous similar messages [ 3146.276282] Lustre: server umount lustre-MDT0000 complete [ 3148.268158] LustreError: 99205:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3148.296870] LustreError: 99205:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3150.418291] LustreError: 99191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789784258 with bad export cookie 16359993926671013944 [ 3150.453874] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3150.878432] Lustre: server umount lustre-MDT0001 complete [ 3164.085682] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 3166.815642] Key type lgssc unregistered [ 3167.115535] LNet: 102748:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3167.120219] LNetError: 102748:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3167.137925] LNet: Removed LNI 192.168.203.145@tcp [ 3167.966282] Key type .llcrypt unregistered [ 3167.968110] Key type ._llcrypt unregistered [ 3184.899329] Key type ._llcrypt registered [ 3184.902150] Key type .llcrypt registered [ 3185.079174] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 3186.410283] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3186.719867] alg: No test for adler32 (adler32-zlib) [ 3188.076476] Lustre: Lustre: Build Version: 2.17.57_110_g5f5d390 [ 3188.392640] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 3190.103319] Key type lgssc registered [ 3192.131526] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3198.207320] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3198.215781] LustreError: 103825:0:(tgt_mount.c:2570:server_fill_super()) cfs_race id 716 sleeping [ 3198.234489] LustreError: 103829:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 waking [ 3198.237220] LustreError: 103825:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 awake: rc=4986 [ 3198.242289] LustreError: 103825:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3198.250987] LustreError: 103825:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3198.256749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3199.705888] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3206.788092] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3207.070047] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3212.544747] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3220.746771] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing lsmod [ 3224.352332] LustreError: 105022:0:(tgt_mount.c:2570:server_fill_super()) cfs_race id 716 sleeping [ 3224.443746] LustreError: 105035:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 waking [ 3224.449779] LustreError: 105022:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 awake: rc=4913 [ 3224.461498] LustreError: 105022:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3224.474739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3224.478097] LustreError: 105022:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3224.910304] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3227.618522] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3227.628618] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3227.652456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3230.181031] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3230.183921] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3230.187678] Lustre: Skipped 1 previous similar message [ 3230.205880] Lustre: Skipped 1 previous similar message [ 3232.295987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3232.309358] Lustre: Skipped 3 previous similar messages [ 3233.872266] Lustre: server umount lustre-MDT0000 complete [ 3237.373750] LustreError: 103867:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3237.402638] LustreError: 103867:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3237.509600] LustreError: 103845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789784345 with bad export cookie 11663782077196533513 [ 3237.525446] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3237.527141] LustreError: 103845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3237.544593] Lustre: 103859:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.145@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 3238.495712] LustreError: 104297:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000001961a677 x1876724857983488/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3238.661062] Lustre: server umount lustre-MDT0001 complete [ 3253.024069] Lustre: server umount lustre-OST0000 complete [ 3263.292395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3263.708289] LustreError: 106099:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3263.818214] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3268.077582] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3269.102538] LustreError: 106100:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3273.187873] LustreError: 106099:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3276.901855] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3277.240295] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3283.705758] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3292.153691] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3297.707907] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3305.263947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3305.577925] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3308.651379] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3311.185637] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3318.614222] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3329.505044] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3329.511093] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3329.531980] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3333.608579] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3333.610959] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3335.701518] Lustre: server umount lustre-OST0000 complete [ 3343.844045] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3343.850494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3343.869118] Lustre: Skipped 1 previous similar message [ 3343.908935] Lustre: Skipped 2 previous similar messages [ 3347.444337] Lustre: server umount lustre-MDT0000 complete [ 3348.961505] LustreError: 106096:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3349.000517] LustreError: 106096:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3351.531489] LustreError: 106079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789784459 with bad export cookie 11663782077196534192 [ 3351.555041] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3351.556418] LustreError: 106079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3351.909787] Lustre: server umount lustre-MDT0001 complete [ 3365.435357] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 3368.925085] Key type lgssc unregistered [ 3369.170832] LNet: 109632:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3369.193559] LNetError: 109632:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3369.223957] LNet: Removed LNI 192.168.203.145@tcp [ 3370.240809] Key type .llcrypt unregistered [ 3370.245172] Key type ._llcrypt unregistered [ 3397.308814] Key type ._llcrypt registered [ 3397.313319] Key type .llcrypt registered [ 3397.462085] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 3398.149544] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3398.185544] alg: No test for adler32 (adler32-zlib) [ 3399.390767] Lustre: Lustre: Build Version: 2.17.57_110_g5f5d390 [ 3399.629746] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 3401.282507] Key type lgssc registered [ 3402.651638] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3410.230851] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3410.253835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3412.005892] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3416.946687] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3421.162704] LustreError: 111044:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3421.196603] LustreError: 111044:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3426.276903] LustreError: 111045:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3427.180423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3427.762355] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3434.278554] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3442.369788] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3448.796026] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3456.241220] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3456.486983] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 3456.647360] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3462.150563] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3464.212701] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3472.664714] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3479.272332] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3484.324856] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3492.835443] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3492.835745] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3492.855579] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3492.871746] LustreError: Skipped 1 previous similar message [ 3492.895565] Lustre: Skipped 1 previous similar message [ 3494.683739] Lustre: server umount lustre-OST0000 complete [ 3499.495021] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3499.510986] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3499.526626] Lustre: Skipped 1 previous similar message [ 3499.530859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3503.075348] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3503.075776] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3508.199404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3508.206424] Lustre: Skipped 2 previous similar messages [ 3513.311203] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3513.497651] Lustre: server umount lustre-MDT0000 complete [ 3517.410089] LustreError: 111045:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3517.424068] LustreError: 111045:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3517.483578] LustreError: 111024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789784625 with bad export cookie 15102417909225546734 [ 3517.490706] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3517.494128] LustreError: 111024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3517.874148] Lustre: server umount lustre-MDT0001 complete [ 3529.335664] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 3532.313386] Key type lgssc unregistered [ 3532.680609] LNet: 114685:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3532.689567] LNetError: 114685:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3532.719982] LNet: Removed LNI 192.168.203.145@tcp [ 3533.603984] Key type .llcrypt unregistered [ 3533.607687] Key type ._llcrypt unregistered [ 3550.063289] Key type ._llcrypt registered [ 3550.065836] Key type .llcrypt registered [ 3550.157984] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 3551.083923] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3551.116886] alg: No test for adler32 (adler32-zlib) [ 3552.190246] Lustre: Lustre: Build Version: 2.17.57_110_g5f5d390 [ 3552.369605] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 3554.076100] Key type lgssc registered [ 3555.464815] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3564.162297] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3564.177902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3566.068402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3570.752889] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3575.266083] LustreError: 115902:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3575.288771] LustreError: 115902:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3580.399949] LustreError: 115903:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3581.087297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3581.330095] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3585.046280] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3593.473410] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3599.595140] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3606.568981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3606.877235] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3610.992463] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3612.324231] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3620.774842] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3631.587414] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3631.620888] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3631.623584] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3631.636936] Lustre: Skipped 1 previous similar message [ 3636.706792] LustreError: 117725:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3636.726188] LustreError: 117725:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3636.886262] Lustre: server umount lustre-OST0000 complete [ 3641.827852] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3641.848640] Lustre: Skipped 2 previous similar messages [ 3641.853233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3646.945090] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3646.953846] Lustre: Skipped 2 previous similar messages [ 3647.618516] Lustre: server umount lustre-MDT0000 complete [ 3651.011455] LustreError: 115882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789784759 with bad export cookie 7495340019184135007 [ 3651.016071] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3651.019563] LustreError: 115882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3651.383765] Lustre: server umount lustre-MDT0001 complete [ 3663.732225] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 3667.319298] Key type lgssc unregistered [ 3667.737246] LNet: 119433:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3667.752084] LNetError: 119433:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3667.775071] LNet: Removed LNI 192.168.203.145@tcp [ 3668.894503] Key type .llcrypt unregistered [ 3668.897286] Key type ._llcrypt unregistered [ 3695.551371] Key type ._llcrypt registered [ 3695.555534] Key type .llcrypt registered [ 3695.790499] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 3696.787439] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3696.827305] alg: No test for adler32 (adler32-zlib) [ 3697.883897] Lustre: Lustre: Build Version: 2.17.57_110_g5f5d390 [ 3698.109901] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 3699.743221] Key type lgssc registered [ 3700.890673] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3709.050551] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3709.080058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3710.625415] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3715.185794] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3720.182379] LustreError: 120846:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3720.218417] LustreError: 120846:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3723.155205] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3723.577392] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3728.185378] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3737.943894] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3746.886441] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3754.513643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3754.918722] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3755.981764] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3760.877302] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3768.257280] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3774.718126] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3779.116837] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3779.132443] Lustre: Skipped 2 previous similar messages [ 3781.284160] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3781.320666] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3785.309851] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3785.336889] Lustre: Skipped 2 previous similar messages [ 3791.031485] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3791.046169] Lustre: Skipped 1 previous similar message [ 3797.560923] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.45@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3797.572877] Lustre: Skipped 2 previous similar messages [ 3801.611821] Lustre: lustre-MDT0000: nosquash_nids set to 2@gni 0@lo 192.168.203.45@tcp 192.168.0.[2,10]@tcp [ 3810.874452] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.45/24@tcp in log lustre-MDT0000 [ 3810.880797] Lustre: Skipped 2 previous similar messages [ 3821.123698] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.45/24@tcp [ 3821.132206] Lustre: Skipped 1 previous similar message [ 3827.171746] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3827.176480] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3827.189207] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3828.516883] Lustre: server umount lustre-OST0000 complete [ 3835.875555] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3835.884934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3838.334336] Lustre: server umount lustre-MDT0000 complete [ 3840.997240] LustreError: 120847:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3841.015159] LustreError: 120847:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3841.777606] LustreError: 124185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789784949 with bad export cookie 12912675955011407054 [ 3841.779159] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3841.795222] LustreError: 124185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3851.436607] Lustre: server umount lustre-MDT0001 complete [ 3864.022960] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 3866.768214] Key type lgssc unregistered [ 3867.037620] LNet: 125793:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3867.058403] LNetError: 125793:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3867.071782] LNet: Removed LNI 192.168.203.145@tcp [ 3867.862493] Key type .llcrypt unregistered [ 3867.868765] Key type ._llcrypt unregistered [ 3901.751564] Key type ._llcrypt registered [ 3901.755535] Key type .llcrypt registered [ 3901.869680] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 3902.673546] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3902.717643] alg: No test for adler32 (adler32-zlib) [ 3903.899356] Lustre: Lustre: Build Version: 2.17.57_110_g5f5d390 [ 3904.100917] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 3905.752822] Key type lgssc registered [ 3906.995795] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3914.216838] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3914.227367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3915.749036] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3915.789424] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.45/24@tcp [ 3915.829079] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3920.773621] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3926.006570] LustreError: 127360:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3926.032607] LustreError: 127360:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3929.593513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3929.907162] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3929.933632] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.45/24@tcp [ 3929.985562] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3934.892060] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3943.106324] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3949.260754] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3956.386487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3956.928871] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3961.026399] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3962.991908] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3971.057174] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3984.864860] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3984.873108] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3984.879537] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3984.903364] Lustre: Skipped 1 previous similar message [ 3989.988341] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3989.997071] Lustre: Skipped 2 previous similar messages [ 3990.620708] Lustre: server umount lustre-OST0000 complete [ 3995.106905] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3995.115130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3995.135663] Lustre: Skipped 1 previous similar message [ 4000.224239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4000.232795] Lustre: Skipped 2 previous similar messages [ 4000.699830] Lustre: server umount lustre-MDT0000 complete [ 4004.411033] LustreError: 129207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789785112 with bad export cookie 4076904453084497893 [ 4004.416175] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4004.420451] LustreError: 129207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4004.764579] Lustre: server umount lustre-MDT0001 complete [ 4015.116772] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 4018.065404] Key type lgssc unregistered [ 4018.347722] LNet: 130952:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4018.362890] LNetError: 130952:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4018.382953] LNet: Removed LNI 192.168.203.145@tcp [ 4019.406767] Key type .llcrypt unregistered [ 4019.408419] Key type ._llcrypt unregistered [ 4053.206682] Key type ._llcrypt registered [ 4053.209371] Key type .llcrypt registered [ 4053.326590] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 4071.321829] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 4072.725204] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4072.877555] alg: No test for adler32 (adler32-zlib) [ 4074.020774] Lustre: Lustre: Build Version: 2.17.57_110_g5f5d390 [ 4074.277304] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 4075.927274] Key type lgssc registered [ 4077.238306] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4113.410923] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4113.452430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4114.821765] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4114.918288] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4115.094532] Lustre: lustre-MDT0000: new disk, initializing [ 4115.176486] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4115.193533] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4120.660605] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4131.774331] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4131.901482] Lustre: 134356:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4131.965612] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4131.970201] Lustre: Skipped 1 previous similar message [ 4132.078489] Lustre: lustre-MDT0001: new disk, initializing [ 4132.180416] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4132.212222] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4132.227554] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4136.658045] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4145.729642] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4150.867697] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4158.673913] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4158.933617] Lustre: lustre-OST0000: new disk, initializing [ 4158.938100] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4158.949753] Lustre: 136324:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4159.108676] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4165.273654] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4165.289795] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4165.419947] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4166.928566] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4177.299343] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4181.251915] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4181.546604] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4186.215151] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4186.474532] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4191.201527] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4191.207219] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4191.215986] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4191.217334] LustreError: Skipped 1 previous similar message [ 4191.239016] Lustre: Skipped 1 previous similar message [ 4195.585827] Lustre: server umount lustre-OST0000 complete [ 4201.440153] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4201.443733] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4201.450833] Lustre: Skipped 1 previous similar message [ 4201.458064] Lustre: Skipped 1 previous similar message [ 4205.878893] Lustre: server umount lustre-MDT0000 complete [ 4209.353695] LustreError: 134346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789785317 with bad export cookie 8009497957602193469 [ 4209.364496] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4209.367812] LustreError: 134346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4209.791365] Lustre: server umount lustre-MDT0001 complete [ 4218.927432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4219.431859] LustreError: 138544:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4219.453317] LustreError: 138544:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4219.515778] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4224.320459] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4225.002930] LustreError: 138545:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4229.088852] LustreError: 138544:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4233.282284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4233.652399] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4238.590718] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4245.091679] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4250.603797] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4257.634584] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4258.167158] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4266.349627] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4276.890601] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4282.326739] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4282.505973] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4286.850866] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4287.210283] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4295.095768] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4295.307213] Lustre: lustre-OST0001: new disk, initializing [ 4295.313655] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4295.322643] Lustre: 141919:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4295.399443] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4302.656049] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4305.463757] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4305.471731] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4305.604469] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4311.581555] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4311.806165] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4316.870467] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4317.101931] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4322.528913] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8dc58413f000.ost_server_uuid 50 [ 4324.349109] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8dc58413f000.ost_server_uuid in IDLE state after 0 sec [ 4334.303854] Lustre: server umount lustre-OST0001 complete [ 4336.616031] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4336.624388] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4336.639382] LustreError: 140863:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4336.693934] LustreError: Skipped 1 previous similar message [ 4341.218525] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4341.231894] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4341.249781] Lustre: Skipped 1 previous similar message [ 4341.265731] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4341.276136] Lustre: Skipped 1 previous similar message [ 4341.730892] LustreError: 140369:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4341.742273] LustreError: 140369:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4344.246593] Lustre: server umount lustre-OST0000 complete [ 4351.457285] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4351.479739] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4351.492181] Lustre: Skipped 1 previous similar message [ 4351.502222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4351.505024] Lustre: Skipped 1 previous similar message [ 4357.088845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4357.094132] Lustre: Skipped 2 previous similar messages [ 4362.207112] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4362.334214] Lustre: server umount lustre-MDT0000 complete [ 4365.570017] LustreError: 138525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789785473 with bad export cookie 8009497957602194351 [ 4365.579244] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4365.581555] LustreError: 138525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4365.972736] Lustre: server umount lustre-MDT0001 complete [ 4384.829762] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 4387.635417] Key type lgssc unregistered [ 4388.013258] LNet: 144923:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4388.023918] LNetError: 144923:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4388.046961] LNet: Removed LNI 192.168.203.145@tcp [ 4388.882468] Key type .llcrypt unregistered [ 4388.885529] Key type ._llcrypt unregistered [ 4422.662888] Key type ._llcrypt registered [ 4422.667043] Key type .llcrypt registered [ 4422.778432] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 4438.965575] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 4440.052048] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4440.279240] alg: No test for adler32 (adler32-zlib) [ 4441.521817] Lustre: Lustre: Build Version: 2.17.57_110_g5f5d390 [ 4441.945745] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 4443.615412] Key type lgssc registered [ 4444.801573] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4482.830360] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4482.866419] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4484.104233] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4484.137867] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4484.243124] Lustre: lustre-MDT0000: new disk, initializing [ 4484.362869] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4484.394535] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4489.036589] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4500.283888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4500.449480] Lustre: 149080:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4500.502794] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4500.507776] Lustre: Skipped 1 previous similar message [ 4500.573204] Lustre: lustre-MDT0001: new disk, initializing [ 4500.684382] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4500.713525] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4500.723597] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4504.286314] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4513.271600] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4519.985307] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4527.864317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4528.069312] Lustre: lustre-OST0000: new disk, initializing [ 4528.073360] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4528.079356] Lustre: 151049:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4528.173527] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4534.348298] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4534.353368] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4534.435180] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4535.363941] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4545.577424] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4551.381199] Lustre: Failing over lustre-OST0000 [ 4551.543318] Lustre: server umount lustre-OST0000 complete [ 4569.569300] Lustre: 147304:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789785661/real 1789785661] req@00000000f3b99ab7 x1876726171864064/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1789785677 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4569.622969] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4570.013052] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4570.363107] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4570.410805] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4570.423822] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4571.424146] Lustre: 147304:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789785663/real 1789785663] req@00000000dcd519b2 x1876726171865088/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789785679 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4571.467083] Lustre: 147304:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4574.451927] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4576.483374] Lustre: 147304:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789785668/real 1789785668] req@000000002e3b43d8 x1876726171866240/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789785684 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4577.346883] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4580.376053] Lustre: lustre-OST0000: Recovery over after 0:06, of 3 clients 3 recovered and 0 were evicted. [ 4580.378830] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4581.483708] Lustre: 147304:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789785673/real 1789785673] req@00000000b1bd897f x1876726171867264/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789785689 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4581.533302] Lustre: 147304:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4582.136418] Lustre: Failing over lustre-MDT0000 [ 4582.337949] Lustre: server umount lustre-MDT0000 complete [ 4582.880259] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4582.889971] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4584.943295] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4584.963500] Lustre: Skipped 1 previous similar message [ 4585.956628] Lustre: 147302:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789785677/real 1789785677] req@0000000079554c3b x1876726171868416/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789785693 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4586.017068] Lustre: 147302:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4590.049152] LustreError: 152302:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4590.077323] LustreError: 152302:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 4595.192754] LustreError: 149092:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4595.247286] LustreError: 149092:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4600.288738] Lustre: 147305:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789785692/real 1789785692] req@000000009f98fe09 x1876726171877632/t0(0) o400->MGC192.168.203.145@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789785708 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4600.291194] LustreError: 150642:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4600.342604] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4600.367249] LustreError: 150642:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4604.255502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4605.865182] LustreError: 149089:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4605.912525] LustreError: 149089:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4610.082421] LustreError: 147301:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000cd5568de x1876726171882368/t0(0) o250->MGC192.168.203.145@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4610.135164] LustreError: 149093:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4610.171477] LustreError: 149093:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 4610.551903] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4610.652870] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4612.010657] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4615.672033] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4615.677320] Lustre: Skipped 1 previous similar message [ 4615.712658] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 4615.767976] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4615.904433] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4625.900912] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4625.925456] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4625.931360] Lustre: Skipped 1 previous similar message [ 4630.923647] Lustre: server umount lustre-OST0000 complete [ 4636.145764] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4636.148336] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4636.149455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4636.180373] Lustre: Skipped 2 previous similar messages [ 4641.248940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4641.259827] Lustre: Skipped 2 previous similar messages [ 4641.564416] Lustre: server umount lustre-MDT0000 complete [ 4646.213574] LustreError: 149071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789785754 with bad export cookie 11028174431714852543 [ 4646.236816] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4646.237830] LustreError: 149071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4646.586510] Lustre: server umount lustre-MDT0001 complete [ 4660.224561] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 4662.561915] Key type lgssc unregistered [ 4662.789365] LNet: 154706:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4662.801923] LNetError: 154706:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4662.823960] LNet: Removed LNI 192.168.203.145@tcp [ 4663.466637] Key type .llcrypt unregistered [ 4663.468956] Key type ._llcrypt unregistered [ 4700.841857] Key type ._llcrypt registered [ 4700.845257] Key type .llcrypt registered [ 4700.952411] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 4702.122402] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4702.198338] alg: No test for adler32 (adler32-zlib) [ 4703.234777] Lustre: Lustre: Build Version: 2.17.57_110_g5f5d390 [ 4703.413526] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 4705.063280] Key type lgssc registered [ 4706.135176] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4713.574710] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4713.601753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4715.144424] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4720.249153] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4725.747559] LustreError: 156223:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4725.764654] LustreError: 156223:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4730.849677] LustreError: 156224:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4731.137484] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4735.954637] LustreError: 156252:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4735.972034] LustreError: 156252:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4735.992740] LustreError: 156252:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 21, retries 0, failed: rc = -5 [ 4736.652103] Lustre: server umount lustre-MDT0000 complete [ 4753.867684] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 4763.897767] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 4802.664660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4802.950659] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4803.001171] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4803.120978] Lustre: lustre-MDT0000: new disk, initializing [ 4803.216872] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4803.245268] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4808.494525] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4819.665208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4819.834797] Lustre: 159556:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4819.929620] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4819.937024] Lustre: Skipped 1 previous similar message [ 4820.044973] Lustre: lustre-MDT0001: new disk, initializing [ 4820.176461] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4820.236936] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4820.266429] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4826.046466] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4836.282635] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4843.897986] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4852.475263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4852.952392] Lustre: lustre-OST0000: new disk, initializing [ 4852.961364] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4852.968599] Lustre: 161528:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4853.134476] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4859.979684] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4859.996291] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4860.171305] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4860.479956] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4872.244637] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4877.534745] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4879.236715] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 22:46:26 (1789785986) [ 5157.349118] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5157.364121] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5157.375688] Lustre: Skipped 1 previous similar message [ 5157.413381] Lustre: Skipped 2 previous similar messages [ 5161.651852] Lustre: server umount lustre-MDT0000 complete [ 5162.472107] LustreError: 161229:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5162.494513] LustreError: 161229:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5165.830885] LustreError: 159548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789786273 with bad export cookie 9918870136449896091 [ 5165.837187] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5165.841790] LustreError: 159548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5167.584190] LustreError: 159564:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5167.585514] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5167.617161] Lustre: Skipped 1 previous similar message [ 5167.624460] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5170.594764] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5175.712129] LustreError: 159564:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5175.713442] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5175.734423] LustreError: 159564:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5180.383212] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5180.738239] Lustre: server umount lustre-MDT0001 complete [ 5191.541590] Lustre: server umount lustre-OST0000 complete [ 5203.002676] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 5213.616423] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 5255.881379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5256.218968] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5256.266160] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5256.371504] Lustre: lustre-MDT0000: new disk, initializing [ 5256.467819] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5256.489960] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5261.378555] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5272.606020] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5272.774776] Lustre: 166196:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5272.872609] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5272.885027] Lustre: Skipped 1 previous similar message [ 5272.986850] Lustre: lustre-MDT0001: new disk, initializing [ 5273.102689] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5273.153952] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5273.167237] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5278.564599] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5289.492339] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5296.248219] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5303.259374] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5303.667292] Lustre: lustre-OST0000: new disk, initializing [ 5303.670995] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5303.678508] Lustre: 168167:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5303.785589] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5310.510829] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5312.587726] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5312.598882] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5312.790208] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5323.098673] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5330.583541] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5330.850532] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5335.184130] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5335.458796] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5338.594985] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5338.595158] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5338.597364] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5338.610701] Lustre: Skipped 1 previous similar message [ 5338.616532] LustreError: Skipped 1 previous similar message [ 5343.735525] LustreError: 168148:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5343.751198] LustreError: 168148:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5344.054626] Lustre: server umount lustre-OST0000 complete [ 5348.833993] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5348.838682] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5348.854466] Lustre: Skipped 1 previous similar message [ 5348.858859] Lustre: Skipped 1 previous similar message [ 5354.486629] Lustre: server umount lustre-MDT0000 complete [ 5357.900887] LustreError: 166188:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789786465 with bad export cookie 9918870136450054921 [ 5357.921211] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5358.252107] Lustre: server umount lustre-MDT0001 complete [ 5367.412800] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 22:54:34 (1789786474) [ 5383.597659] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 5392.952252] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 5427.700631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5427.947198] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5427.968281] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5428.083808] Lustre: lustre-MDT0000: new disk, initializing [ 5428.186729] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5428.224246] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5432.383760] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5444.911030] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5445.077971] Lustre: 172924:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5445.428506] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5449.938933] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5459.834037] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5465.568051] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5473.422535] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5473.769352] Lustre: lustre-OST0000: new disk, initializing [ 5473.776728] Lustre: Skipped 1 previous similar message [ 5473.783520] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5473.795136] Lustre: Skipped 2 previous similar messages [ 5473.803450] Lustre: 174901:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5480.027268] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5480.036614] Lustre: Skipped 1 previous similar message [ 5480.050525] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5480.183854] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5480.689532] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5490.918587] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5500.897532] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5500.907270] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5500.913085] LustreError: Skipped 1 previous similar message [ 5500.913873] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5500.913876] Lustre: Skipped 3 previous similar messages [ 5500.944372] Lustre: Skipped 1 previous similar message [ 5503.413234] Lustre: server umount lustre-OST0000 complete [ 5507.043053] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5511.143507] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5511.154253] Lustre: Skipped 1 previous similar message [ 5521.375460] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5521.607336] Lustre: server umount lustre-MDT0000 complete [ 5525.488388] LustreError: 174900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789786633 with bad export cookie 9918870136450055803 [ 5525.500359] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5525.509519] LustreError: 174900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 5525.919678] Lustre: server umount lustre-MDT0001 complete [ 5534.944279] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 22:57:22 (1789786642) [ 5550.849385] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 5560.454625] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 5599.291416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5599.550678] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5599.577337] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5599.666956] Lustre: lustre-MDT0000: new disk, initializing [ 5599.803450] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5599.810969] Lustre: Skipped 2 previous similar messages [ 5599.845311] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5604.556609] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5615.310464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5615.475672] Lustre: 179229:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5615.841264] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5620.578418] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5630.692761] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5636.382623] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5643.370075] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5643.607250] Lustre: 181204:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5650.059334] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5653.021134] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5653.143610] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5659.294873] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5668.320183] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5668.337616] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5668.351960] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5668.357364] Lustre: Skipped 8 previous similar messages [ 5672.326277] Lustre: server umount lustre-OST0000 complete [ 5677.538692] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5677.552955] LustreError: Skipped 1 previous similar message [ 5682.707755] Lustre: server umount lustre-MDT0000 complete [ 5684.192805] LustreError: 179241:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5684.218512] LustreError: 179241:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5686.844856] LustreError: 179221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789786794 with bad export cookie 9918870136450056888 [ 5686.855921] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5686.858478] LustreError: 179221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5687.293376] Lustre: server umount lustre-MDT0001 complete [ 5697.641328] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 5700.205920] Key type lgssc unregistered [ 5700.518109] LNet: 183061:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5700.524903] LNetError: 183061:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5701.555629] LNet: Removed LNI 192.168.203.145@tcp [ 5702.525327] Key type .llcrypt unregistered [ 5702.529741] Key type ._llcrypt unregistered [ 5724.840747] Key type ._llcrypt registered [ 5724.845686] Key type .llcrypt registered [ 5724.956937] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 5725.833906] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5725.867963] alg: No test for adler32 (adler32-zlib) [ 5726.978415] Lustre: Lustre: Build Version: 2.17.57_110_g5f5d390 [ 5727.221135] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 5728.879489] Key type lgssc registered [ 5729.812870] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5736.260969] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5736.275955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5737.672547] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5741.525823] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5747.175258] LustreError: 184486:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5747.196368] LustreError: 184486:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5749.930191] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5750.335689] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5754.576746] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5761.882123] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5767.364740] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5772.841853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5773.193798] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5778.305106] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5782.463652] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5785.070647] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5802.978618] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5802.983659] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5802.999556] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5803.005071] Lustre: Skipped 1 previous similar message [ 5808.101740] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5808.109819] Lustre: Skipped 1 previous similar message [ 5808.787622] Lustre: server umount lustre-OST0000 complete [ 5813.218314] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5813.220843] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5813.229562] Lustre: Skipped 2 previous similar messages [ 5813.242193] Lustre: Skipped 1 previous similar message [ 5818.338656] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5818.348201] Lustre: Skipped 1 previous similar message [ 5818.705743] Lustre: server umount lustre-MDT0000 complete [ 5822.308893] LustreError: 184467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789786930 with bad export cookie 15756636421192655829 [ 5822.317110] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5822.318576] LustreError: 184467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5822.600624] Lustre: server umount lustre-MDT0001 complete [ 5832.598438] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 5835.164141] Key type lgssc unregistered [ 5835.412700] LNet: 188047:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5835.435425] LNetError: 188047:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5835.456676] LNet: Removed LNI 192.168.203.145@tcp [ 5836.142925] Key type .llcrypt unregistered [ 5836.152697] Key type ._llcrypt unregistered [ 5859.939482] Key type ._llcrypt registered [ 5859.941920] Key type .llcrypt registered [ 5860.016502] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 5861.316582] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5861.356268] alg: No test for adler32 (adler32-zlib) [ 5862.364886] Lustre: Lustre: Build Version: 2.17.57_110_g5f5d390 [ 5862.623349] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 5864.279360] Key type lgssc registered [ 5865.387244] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5874.225919] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5874.250877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5875.736708] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5880.352324] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5885.929594] LustreError: 189463:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5885.943230] LustreError: 189463:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5888.800188] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5889.270520] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5893.924122] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5901.859398] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5908.123417] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5914.248298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5914.610907] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5920.298192] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5920.752150] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5927.820581] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5939.168988] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5939.169838] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5939.172070] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5939.185045] LustreError: Skipped 1 previous similar message [ 5939.213354] Lustre: Skipped 1 previous similar message [ 5940.674725] Lustre: lustre-OST0000: Not available for connect from 192.168.203.45@tcp (stopping) [ 5940.683833] Lustre: Skipped 1 previous similar message [ 5944.288495] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5949.413054] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5949.417718] Lustre: Skipped 2 previous similar messages [ 5951.967158] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5952.096560] Lustre: server umount lustre-OST0000 complete [ 5959.469403] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8dc58324b000.ost_server_uuid 50 [ 5960.841804] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5975.013285] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5975.029933] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5975.039763] Lustre: Skipped 3 previous similar messages [ 5980.819919] Lustre: server umount lustre-MDT0000 complete [ 5984.224957] LustreError: 189458:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5984.246973] LustreError: 189458:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5984.298249] LustreError: 189444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789787092 with bad export cookie 5786354127123836774 [ 5984.300292] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5984.308234] LustreError: 189444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5984.326438] LustreError: 190193:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0001: can't precreate: rc = -5 [ 5984.337653] LustreError: 190193:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0001: cannot precreate objects: rc = -5 [ 5984.631310] Lustre: server umount lustre-MDT0001 complete [ 5992.473523] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 23:04:59 (1789787099) [ 5998.659293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5999.096736] LustreError: 193379:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5999.148371] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6002.937639] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6004.195667] LustreError: 193380:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6009.312632] LustreError: 193379:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6010.477820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6010.731772] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6015.218652] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6023.139769] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6029.105501] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6035.122133] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6035.359429] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6040.986815] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6045.667520] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6049.127982] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6055.348225] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6055.547202] Lustre: lustre-OST0001: new disk, initializing [ 6055.550963] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6055.556302] Lustre: 196222:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6055.665085] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6061.761925] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6065.244611] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6065.265920] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6065.380916] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6071.333895] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6075.877144] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6075.882658] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6075.899146] Lustre: Skipped 2 previous similar messages [ 6075.915477] Lustre: Skipped 3 previous similar messages [ 6080.381822] Lustre: server umount lustre-OST0000 complete [ 6080.999357] LustreError: 195201:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6081.011465] LustreError: 195201:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6086.810456] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6087.037865] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6091.494840] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6091.730860] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6097.831385] LustreError: 195200:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6097.848256] LustreError: 195200:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 6106.593958] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6106.597251] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6106.598961] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6106.611119] Lustre: Skipped 1 previous similar message [ 6106.634448] Lustre: Skipped 2 previous similar messages [ 6111.111375] Lustre: server umount lustre-OST0001 complete [ 6116.835939] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6116.854180] Lustre: Skipped 1 previous similar message [ 6121.284194] Lustre: server umount lustre-MDT0000 complete [ 6125.570307] LustreError: 193359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789787233 with bad export cookie 5786354127123837845 [ 6125.574201] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6125.581623] LustreError: 193359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6125.886863] Lustre: server umount lustre-MDT0001 complete [ 6148.216786] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 23:07:35 (1789787255) [ 6155.263073] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6155.605832] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6155.655206] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6155.859627] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6160.484289] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6168.701363] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6168.817898] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6168.854612] Lustre: 199657:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6173.216368] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6180.227432] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6185.436749] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6192.105236] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6192.348517] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6199.044436] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6202.015060] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6207.084773] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6214.058825] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6214.122073] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6221.570497] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6221.991056] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6230.945798] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6237.130051] Lustre: lustre-OST0000: Not available for connect from 192.168.203.45@tcp (stopping) [ 6237.138413] Lustre: Skipped 2 previous similar messages [ 6237.163257] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6240.001936] Lustre: server umount lustre-OST0000 complete [ 6242.220802] LustreError: 201511:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6242.240304] LustreError: 201511:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 6257.633271] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6257.640359] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6257.645850] LustreError: Skipped 1 previous similar message [ 6257.668450] Lustre: Skipped 2 previous similar messages [ 6261.191281] Lustre: server umount lustre-OST0001 complete [ 6266.336868] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6270.839771] Lustre: server umount lustre-MDT0000 complete [ 6274.927625] LustreError: 202546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789787382 with bad export cookie 5786354127123839252 [ 6274.928540] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6274.948844] LustreError: 202546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6275.348882] Lustre: server umount lustre-MDT0001 complete [ 6303.226614] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 23:10:10 (1789787410) [ 6320.505173] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 6331.851160] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 6364.338284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6364.585746] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6364.615787] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6364.696948] Lustre: lustre-MDT0000: new disk, initializing [ 6364.761862] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6364.767026] Lustre: Skipped 3 previous similar messages [ 6364.782073] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6368.971770] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6379.201314] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6379.412969] Lustre: 207753:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6379.468356] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6379.487432] Lustre: Skipped 1 previous similar message [ 6379.636410] Lustre: lustre-MDT0001: new disk, initializing [ 6379.858615] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6379.873669] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6384.883343] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6394.132357] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6400.094277] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6406.432902] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6406.842945] Lustre: lustre-OST0000: new disk, initializing [ 6406.847126] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6406.853566] Lustre: 209720:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6413.289645] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6415.936715] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6415.944839] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6416.024178] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6422.954663] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6428.336651] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6428.504210] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6433.562783] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6433.962220] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6436.843312] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6436.848153] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6436.848349] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6436.848352] Lustre: Skipped 7 previous similar messages [ 6436.862525] Lustre: Skipped 3 previous similar messages [ 6442.146348] Lustre: server umount lustre-OST0000 complete [ 6446.570682] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6452.543357] Lustre: server umount lustre-MDT0000 complete [ 6456.146090] LustreError: 207744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789787564 with bad export cookie 5786354127123841037 [ 6456.147695] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6456.157763] LustreError: 207744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6456.750233] Lustre: server umount lustre-MDT0001 complete [ 6465.473249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6466.080265] LustreError: 211938:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6466.125120] LustreError: 211938:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 6470.424449] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6478.265648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6482.440262] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6489.755144] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6494.882080] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6501.182835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6501.444386] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6501.449146] Lustre: Skipped 4 previous similar messages [ 6506.510472] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6513.662712] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6518.383721] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6518.649818] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6524.038681] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6524.305450] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6527.463426] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6527.464596] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6527.479929] Lustre: Skipped 2 previous similar messages [ 6532.671382] Lustre: server umount lustre-OST0000 complete [ 6539.340163] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6544.448704] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6573.069054] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6579.562302] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6582.629931] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6582.655100] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6582.659535] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6582.685970] Lustre: Skipped 1 previous similar message [ 6586.712787] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6591.636798] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 3 sec [ 6598.120291] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6598.120744] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6598.140904] Lustre: Skipped 13 previous similar messages [ 6601.501909] Lustre: server umount lustre-OST0000 complete [ 6611.525898] Lustre: server umount lustre-MDT0000 complete [ 6613.475784] LustreError: 211938:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6613.503405] LustreError: 211938:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6615.213954] LustreError: 211920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789787723 with bad export cookie 5786354127123841912 [ 6615.217879] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6615.224897] LustreError: 211920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6615.561139] Lustre: server umount lustre-MDT0001 complete [ 6623.966582] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 23:15:31 (1789787731) [ 6630.614121] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6635.710657] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6644.446512] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6648.993667] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6656.511657] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6662.203553] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6670.454906] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6677.670831] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6685.430941] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6691.725690] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6692.130525] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6697.652650] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6697.996693] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6706.907496] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6707.294811] Lustre: lustre-OST0001: new disk, initializing [ 6707.298533] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6707.305568] Lustre: 221275:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6714.716688] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6716.150359] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6716.164572] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6716.354453] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6726.833308] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6732.503523] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6732.756785] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6737.628095] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6737.853602] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6742.008265] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6742.008325] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6742.017085] LustreError: Skipped 3 previous similar messages [ 6742.030951] Lustre: Skipped 6 previous similar messages [ 6754.271728] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6754.463553] Lustre: server umount lustre-OST0001 complete [ 6760.277765] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6760.481765] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6765.724518] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6766.037584] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6795.905101] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6796.227476] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6796.238033] Lustre: Skipped 5 previous similar messages [ 6801.398104] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6801.432730] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6801.443976] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6803.384453] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6811.798329] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6813.621337] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6821.869406] Lustre: server umount lustre-OST0001 complete [ 6853.087338] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6853.334448] Lustre: server umount lustre-OST0000 complete [ 6857.706105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6857.715249] Lustre: Skipped 19 previous similar messages [ 6863.577589] Lustre: server umount lustre-MDT0000 complete [ 6868.020108] LustreError: 217888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789787976 with bad export cookie 5786354127123842885 [ 6868.024785] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6868.035833] LustreError: 217888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6874.407657] Lustre: server umount lustre-MDT0001 complete [ 6902.870484] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 23:20:09 (1789788009) [ 6910.456333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6910.999714] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6911.047357] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6916.909490] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6926.988348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6927.086990] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6927.126360] Lustre: 226525:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6932.907872] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6941.360440] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6948.563780] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6955.406193] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6955.541120] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6961.550690] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6969.084448] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6978.244424] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6978.307303] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6984.955888] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6992.069981] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6993.739937] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6998.253313] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6998.449599] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7002.828150] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7003.005902] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7008.268880] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8dc5913e9000.ost_server_uuid 50 [ 7009.764453] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8dc5913e9000.ost_server_uuid in IDLE state after 0 sec [ 7010.942455] Lustre: Permanently deactivating lustre-OST0001 [ 7010.978786] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7014.446762] Lustre: Permanently reactivating lustre-OST0001 [ 7021.026400] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7021.040987] LustreError: Skipped 3 previous similar messages [ 7021.050941] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7021.067789] Lustre: Skipped 6 previous similar messages [ 7024.910992] Lustre: server umount lustre-OST0001 complete [ 7026.155710] LustreError: 228370:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7026.177173] LustreError: 228370:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 42 previous similar messages [ 7035.338909] Lustre: server umount lustre-OST0000 complete [ 7045.805507] Lustre: server umount lustre-MDT0000 complete [ 7050.358635] LustreError: 226517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789788158 with bad export cookie 5786354127123844173 [ 7050.365781] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7050.369629] LustreError: 226517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7050.752147] Lustre: server umount lustre-MDT0001 complete [ 7077.197655] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 23:23:03 (1789788183) [ 7086.701255] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7087.212903] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7087.286390] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7087.320315] Lustre: Skipped 5 previous similar messages [ 7092.892648] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 7101.753648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7101.934119] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7101.987222] Lustre: 233221:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7107.693553] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 7116.123309] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7122.553486] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7129.508849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7129.724661] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7129.766618] Lustre: Permanently deactivating lustre-OST0000 [ 7129.805384] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7136.294349] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 7139.883847] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7143.922597] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7151.614051] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7151.743237] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7157.472112] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7159.662289] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 7168.421928] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7170.547548] Lustre: Permanently reactivating lustre-OST0000 [ 7170.582211] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7170.594549] Lustre: Skipped 2 previous similar messages [ 7176.819287] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7176.822083] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 36 seconds. I think it's dead, and I am evicting it. exp 00000000d42dbb6f, cur 1789788284 deadline 1789788278 last 1789788248 [ 7176.830176] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7176.830690] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7176.830693] Lustre: Skipped 1 previous similar message [ 7176.896109] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7195.013216] Lustre: server umount lustre-OST0001 complete [ 7204.763270] Lustre: server umount lustre-OST0000 complete [ 7223.263136] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7223.707740] Lustre: server umount lustre-MDT0000 complete [ 7228.498555] LustreError: 233973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789788336 with bad export cookie 5786354127123845853 [ 7228.525790] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7229.264683] Lustre: server umount lustre-MDT0001 complete [ 7244.918520] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 7247.571761] Key type lgssc unregistered [ 7247.759653] LNet: 238104:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7247.772630] LNetError: 238104:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7247.789298] LNet: Removed LNI 192.168.203.145@tcp [ 7248.808569] Key type .llcrypt unregistered [ 7248.810892] Key type ._llcrypt unregistered [ 7274.205888] Key type ._llcrypt registered [ 7274.210381] Key type .llcrypt registered [ 7274.344510] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 7275.654974] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7275.752078] alg: No test for adler32 (adler32-zlib) [ 7276.779174] Lustre: Lustre: Build Version: 2.17.57_110_g5f5d390 [ 7277.115940] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 7278.783756] Key type lgssc registered [ 7280.029508] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7290.846984] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7290.887032] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7292.451108] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7297.027583] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 7302.645987] LustreError: 239565:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7302.677850] LustreError: 239565:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7306.106784] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7306.296981] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7306.325740] Lustre: 239553:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7306.334894] Lustre: Permanently deactivating lustre-MDT0001 [ 7306.344477] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7306.659881] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7311.760926] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 7316.494302] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7319.695119] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7325.495585] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7331.646571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7331.902668] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7332.905243] LustreError: 241409:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7332.919331] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:193) [ 7336.977598] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 7337.961374] LustreError: 241409:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7339.007856] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7343.880517] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7344.107311] LustreError: 241409:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7344.131804] LustreError: 241409:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7349.228872] LustreError: 241406:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7349.257256] LustreError: 241406:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7351.850750] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7352.211444] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7357.443171] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7357.463194] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7361.460357] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 7371.563681] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7375.566822] Lustre: Permanently reactivating lustre-MDT0001 [ 7375.579829] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7375.584353] Lustre: Skipped 2 previous similar messages [ 7379.458373] Lustre: lustre-MDT0001: Client 160d77fd-76b9-4748-a318-b46b4b4f6afd (at 192.168.203.45@tcp) reconnecting [ 7383.203854] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7383.229081] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7383.239811] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7383.258042] LustreError: 243245:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 7383.260648] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7383.265505] LustreError: 243245:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 7383.265642] LustreError: 243245:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.145@tcp: cfg command failed: rc = -108 [ 7383.265888] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 7383.265888] [ 7383.268409] LustreError: 239557:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 7386.597788] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7388.185819] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7394.497889] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7394.804211] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7400.067190] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7400.301994] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7403.185975] Lustre: Permanently deactivating lustre-MDT0001 [ 7403.215264] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7403.225820] Lustre: Skipped 2 previous similar messages [ 7411.344491] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7424.480714] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7424.489399] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7424.510302] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7425.012091] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7425.024474] Lustre: Skipped 2 previous similar messages [ 7427.148427] Lustre: server umount lustre-MDT0000 complete [ 7430.122573] LustreError: 239565:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7430.149471] LustreError: 239565:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7430.462078] LustreError: 239547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789788538 with bad export cookie 8041430880096134751 [ 7430.469696] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7430.474021] LustreError: 239547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7430.835452] Lustre: server umount lustre-MDT0001 complete [ 7436.507359] Lustre: server umount lustre-OST0000 complete [ 7448.950798] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:29:16 (1789788556) [ 7460.213322] Lustre: server umount lustre-OST0001 complete [ 7467.502778] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 7478.611433] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 7515.834321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7516.019269] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7516.031635] Lustre: Skipped 2 previous similar messages [ 7516.054312] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7516.141578] Lustre: lustre-MDT0000: new disk, initializing [ 7516.217536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7516.238373] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7520.336898] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 7530.983376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7531.069727] Lustre: 247846:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7531.101699] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7531.113211] Lustre: Skipped 1 previous similar message [ 7531.179499] Lustre: lustre-MDT0001: new disk, initializing [ 7531.229236] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7531.248868] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7531.256157] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7534.438876] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 7544.067384] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7551.770791] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7559.119218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7559.437850] Lustre: lustre-OST0000: new disk, initializing [ 7559.443291] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7559.451571] Lustre: 249816:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7559.632517] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7565.401745] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7565.406835] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7565.496840] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7566.453362] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 7576.035117] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7578.364795] LustreError: 249086:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7586.653376] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7586.856882] Lustre: lustre-OST0001: new disk, initializing [ 7586.865734] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7586.879401] Lustre: 250930:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7588.423246] LustreError: 249086:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7588.439234] LustreError: 247853:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7593.180745] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 7596.638288] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7596.656833] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7596.749031] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7598.527765] LustreError: 247853:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7602.012289] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7603.359544] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7607.266980] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7607.277647] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7607.280345] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7607.280557] Lustre: Skipped 4 previous similar messages [ 7611.438628] Lustre: server umount lustre-OST0001 complete [ 7612.397737] LustreError: 249798:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7612.418525] LustreError: 249798:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7632.352659] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7632.356517] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7632.365921] LustreError: Skipped 1 previous similar message [ 7632.366783] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7632.366787] Lustre: Skipped 2 previous similar messages [ 7632.397694] Lustre: Skipped 1 previous similar message [ 7635.119788] Lustre: server umount lustre-OST0000 complete [ 7639.008102] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7639.019208] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7639.031085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7639.035984] Lustre: Skipped 1 previous similar message [ 7648.228588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7648.242379] Lustre: Skipped 3 previous similar messages [ 7653.343531] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7653.568089] Lustre: server umount lustre-MDT0000 complete [ 7657.157076] LustreError: 248642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789788765 with bad export cookie 8041430880096137789 [ 7657.180981] LustreError: 248642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7657.181131] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7663.760408] Lustre: server umount lustre-MDT0001 complete [ 7673.872929] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 7676.355605] Key type lgssc unregistered [ 7676.643207] LNet: 252964:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7676.652432] LNetError: 252964:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7676.676177] LNet: Removed LNI 192.168.203.145@tcp [ 7677.453455] Key type .llcrypt unregistered [ 7677.456081] Key type ._llcrypt unregistered [ 7715.483725] Key type ._llcrypt registered [ 7715.494846] Key type .llcrypt registered [ 7715.603845] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 7716.759962] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7716.810538] alg: No test for adler32 (adler32-zlib) [ 7717.864285] Lustre: Lustre: Build Version: 2.17.57_110_g5f5d390 [ 7718.083619] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 7719.743686] Key type lgssc registered [ 7721.002789] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7729.087235] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7729.117670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7730.347588] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7730.370696] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7730.529629] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7734.056834] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 7741.721262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7741.801846] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7741.824746] Lustre: 255167:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7742.081133] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7746.878882] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 7755.954235] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7762.041829] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7768.800663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7768.951061] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7769.111642] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7775.957609] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 7777.481636] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7784.053656] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7839.203747] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7839.212779] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7839.223436] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7839.229317] Lustre: Skipped 1 previous similar message [ 7842.454110] Lustre: server umount lustre-OST0000 complete [ 7846.239343] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7857.095766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7857.508668] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7859.571480] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7859.592346] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7859.594609] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7859.624212] Lustre: Skipped 1 previous similar message [ 7864.140535] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 7871.381555] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7915.439099] Lustre: server umount lustre-OST0000 complete [ 7921.122068] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7921.123947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7921.124955] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7921.124959] LustreError: Skipped 1 previous similar message [ 7921.128503] Lustre: Skipped 2 previous similar messages [ 7921.150327] Lustre: Skipped 1 previous similar message [ 7926.244321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7926.250507] Lustre: Skipped 1 previous similar message [ 7929.503870] Lustre: server umount lustre-MDT0000 complete [ 7932.383171] Lustre: 254488:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789789024/real 1789789024] req@00000000fe062750 x1876729608083456/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789789040 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7932.387436] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7932.405797] Lustre: 254488:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 7933.620721] LustreError: 255157:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789789041 with bad export cookie 3130937324486764733 [ 7933.622102] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7933.625030] LustreError: 255157:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7933.853153] Lustre: server umount lustre-MDT0001 complete [ 7945.365747] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 7948.332613] Key type lgssc unregistered [ 7948.590169] LNet: 260590:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7948.600888] LNetError: 260590:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7948.629555] LNet: Removed LNI 192.168.203.145@tcp [ 7949.500621] Key type .llcrypt unregistered [ 7949.503310] Key type ._llcrypt unregistered [ 7973.316227] Key type ._llcrypt registered [ 7973.318188] Key type .llcrypt registered [ 7973.431488] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 7974.701120] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7974.769535] alg: No test for adler32 (adler32-zlib) [ 7975.911605] Lustre: Lustre: Build Version: 2.17.57_110_g5f5d390 [ 7976.176759] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 7977.879186] Key type lgssc registered [ 7979.236664] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7986.913872] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7986.938939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7988.389236] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7992.747883] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 7998.946424] LustreError: 262002:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7998.965782] LustreError: 262002:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8001.452565] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8001.778025] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8005.336623] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 8012.580386] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8019.057832] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8028.432088] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8028.823977] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8033.967996] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8034.471991] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 8042.432317] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8070.113465] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8070.123284] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8070.139724] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8072.162527] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8072.168435] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8072.172987] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8076.269635] Lustre: server umount lustre-OST0000 complete [ 8082.408151] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8082.412706] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8082.431343] Lustre: Skipped 1 previous similar message [ 8082.458828] Lustre: Skipped 2 previous similar messages [ 8086.379762] Lustre: server umount lustre-MDT0000 complete [ 8087.521039] LustreError: 261998:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8087.538917] LustreError: 261998:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8089.725638] LustreError: 261982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789789197 with bad export cookie 10498018656119269570 [ 8089.729415] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8089.736037] LustreError: 261982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8090.032842] Lustre: server umount lustre-MDT0001 complete [ 8100.209449] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 8102.508665] Key type lgssc unregistered [ 8102.803769] LNet: 266308:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8102.809953] LNetError: 266308:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8102.834180] LNet: Removed LNI 192.168.203.145@tcp [ 8103.737892] Key type .llcrypt unregistered [ 8103.741491] Key type ._llcrypt unregistered [ 8123.778470] Key type ._llcrypt registered [ 8123.786505] Key type .llcrypt registered [ 8123.948384] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 8125.033130] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8125.196988] alg: No test for adler32 (adler32-zlib) [ 8126.318784] Lustre: Lustre: Build Version: 2.17.57_110_g5f5d390 [ 8126.591410] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 8128.295168] Key type lgssc registered [ 8129.607536] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8137.879167] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8137.914494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8139.444845] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8144.046155] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 8148.970906] LustreError: 267530:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8149.005241] LustreError: 267530:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8153.435453] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8153.711830] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8157.835887] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 8164.812425] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8170.127878] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8177.109021] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8177.464376] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8184.426095] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 8184.623364] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8192.507734] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8205.285668] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8205.306160] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8205.333435] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8210.407098] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8210.416541] Lustre: Skipped 1 previous similar message [ 8211.105997] Lustre: server umount lustre-OST0000 complete [ 8215.536644] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8215.545203] Lustre: Skipped 1 previous similar message [ 8215.550166] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8215.558288] Lustre: Skipped 1 previous similar message [ 8220.128347] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8220.135233] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8220.154848] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8221.545334] Lustre: server umount lustre-MDT0000 complete [ 8225.761979] LustreError: 268584:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8225.800578] LustreError: 268584:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8225.930689] LustreError: 267510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789789333 with bad export cookie 9348600694945058202 [ 8225.934709] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8225.939465] LustreError: 267510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8226.365826] Lustre: server umount lustre-MDT0001 complete [ 8236.494954] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 8239.419639] Key type lgssc unregistered [ 8239.822697] LNet: 271208:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8239.828869] LNetError: 271208:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8240.880547] LNet: Removed LNI 192.168.203.145@tcp [ 8241.824184] Key type .llcrypt unregistered [ 8241.828533] Key type ._llcrypt unregistered [ 8262.641704] Key type ._llcrypt registered [ 8262.644484] Key type .llcrypt registered [ 8262.745785] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 8263.361233] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8263.483171] alg: No test for adler32 (adler32-zlib) [ 8264.511551] Lustre: Lustre: Build Version: 2.17.57_110_g5f5d390 [ 8264.717165] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 8266.367251] Key type lgssc registered [ 8267.270635] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8273.189989] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8273.209486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8274.616872] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8278.471649] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 8284.129762] LustreError: 272600:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8284.153613] LustreError: 272600:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8286.451949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8286.756766] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8290.355431] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 8297.410920] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8303.175562] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8308.512108] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8308.749623] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8314.220224] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 8319.051977] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8321.192541] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8342.498407] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8342.506952] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8342.527742] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8344.544722] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8344.556094] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8344.559920] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8344.598454] Lustre: Skipped 1 previous similar message [ 8348.030762] Lustre: server umount lustre-OST0000 complete [ 8353.249448] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8353.254234] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8353.267873] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8357.765564] Lustre: server umount lustre-MDT0000 complete [ 8359.905706] LustreError: 273670:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8359.935147] LustreError: 273670:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8361.538771] LustreError: 272580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789789469 with bad export cookie 15051270490244119581 [ 8361.555047] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8361.943708] Lustre: server umount lustre-MDT0001 complete [ 8371.326079] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 8373.753863] Key type lgssc unregistered [ 8374.058777] LNet: 276904:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8374.068145] LNetError: 276904:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8374.088360] LNet: Removed LNI 192.168.203.145@tcp [ 8374.834545] Key type .llcrypt unregistered [ 8374.837548] Key type ._llcrypt unregistered [ 8392.484661] Key type ._llcrypt registered [ 8392.487109] Key type .llcrypt registered [ 8392.584320] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 8393.506591] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8393.602717] alg: No test for adler32 (adler32-zlib) [ 8394.746381] Lustre: Lustre: Build Version: 2.17.57_110_g5f5d390 [ 8394.988057] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 8396.719185] Key type lgssc registered [ 8397.707556] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8405.254902] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8405.268402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8406.861237] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8411.525423] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 8416.227738] LustreError: 278137:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8416.254982] LustreError: 278137:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8419.888974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8420.287556] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8425.234333] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 8432.594636] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8437.571592] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8444.301092] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8444.540812] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8449.966236] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 8451.699615] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8456.516286] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8466.402684] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8466.423278] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8466.466182] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8467.427684] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8467.433867] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8467.444093] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8471.625758] Lustre: server umount lustre-OST0000 complete [ 8476.640866] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8476.667114] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8476.708825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8476.711670] Lustre: Skipped 1 previous similar message [ 8482.784557] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8482.792241] Lustre: Skipped 3 previous similar messages [ 8483.132838] Lustre: server umount lustre-MDT0000 complete [ 8487.253150] LustreError: 278102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789789595 with bad export cookie 7552753642547347950 [ 8487.272285] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8487.275075] LustreError: 278102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8487.725760] Lustre: server umount lustre-MDT0001 complete [ 8499.394336] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 8502.126265] Key type lgssc unregistered [ 8502.397409] LNet: 281837:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8502.404617] LNetError: 281837:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8503.463232] LNet: Removed LNI 192.168.203.145@tcp [ 8504.337255] Key type .llcrypt unregistered [ 8504.338777] Key type ._llcrypt unregistered [ 8515.110240] Key type ._llcrypt registered [ 8515.113490] Key type .llcrypt registered [ 8515.237066] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8531.527916] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 8545.817619] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 8546.823089] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8546.952709] alg: No test for adler32 (adler32-zlib) [ 8548.027370] Lustre: Lustre: Build Version: 2.17.57_110_g5f5d390 [ 8548.339259] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 8550.031283] Key type lgssc registered [ 8551.069793] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8584.345089] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8584.372296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8585.674611] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8585.706625] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8585.836898] Lustre: lustre-MDT0000: new disk, initializing [ 8585.929217] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8585.951470] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8589.829189] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 8599.425506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8599.555859] Lustre: 285386:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8599.639185] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8599.644193] Lustre: Skipped 1 previous similar message [ 8599.721192] Lustre: lustre-MDT0001: new disk, initializing [ 8599.816808] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8599.852374] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8599.863659] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8604.209535] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 8613.259681] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8618.621458] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8624.593166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8624.801315] Lustre: lustre-OST0000: new disk, initializing [ 8624.804693] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8624.813262] Lustre: 287354:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8624.903284] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8629.667319] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 8633.942627] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8633.958652] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8634.058820] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8638.750694] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8643.467521] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8643.678342] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8647.844152] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8648.001141] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8649.699133] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8649.707147] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8649.710716] LustreError: Skipped 1 previous similar message [ 8649.711618] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8649.736294] Lustre: Skipped 1 previous similar message [ 8654.825083] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8654.841244] Lustre: Skipped 2 previous similar messages [ 8655.715617] Lustre: server umount lustre-OST0000 complete [ 8659.940047] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8659.962442] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8659.967439] Lustre: Skipped 1 previous similar message [ 8665.016225] Lustre: server umount lustre-MDT0000 complete [ 8669.241350] LustreError: 285379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789789777 with bad export cookie 1660467793772381470 [ 8669.249279] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8669.251881] LustreError: 285379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8669.507688] Lustre: server umount lustre-MDT0001 complete [ 8677.983832] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 23:49:45 (1789789785) [ 8684.742043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8685.319055] LustreError: 289767:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8685.336910] LustreError: 289767:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8685.412353] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8690.645944] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 8690.663288] LustreError: 289768:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8695.779329] LustreError: 289767:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8699.755928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8700.413858] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8705.484990] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 8713.008341] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8718.502626] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8724.343722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8724.606608] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8730.530640] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 8737.315865] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8751.593996] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8751.599572] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8751.604665] Lustre: Skipped 2 previous similar messages [ 8751.607942] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8751.631019] Lustre: Skipped 1 previous similar message [ 8752.751565] Lustre: server umount lustre-OST0000 complete [ 8756.705142] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8756.709630] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8756.720017] Lustre: Skipped 1 previous similar message [ 8756.737576] Lustre: Skipped 1 previous similar message [ 8762.604739] Lustre: server umount lustre-MDT0000 complete [ 8765.856680] LustreError: 289748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789789873 with bad export cookie 1660467793772382352 [ 8765.862452] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8765.870031] LustreError: 289748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8766.129169] Lustre: server umount lustre-MDT0001 complete [ 8777.869513] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 8781.066224] Key type lgssc unregistered [ 8781.419046] LNet: 293301:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8781.436836] LNetError: 293301:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8781.453990] LNet: Removed LNI 192.168.203.145@tcp [ 8782.752685] Key type .llcrypt unregistered [ 8782.761224] Key type ._llcrypt unregistered [ 8812.834326] Key type ._llcrypt registered [ 8812.840377] Key type .llcrypt registered [ 8813.016706] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 8828.206902] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 8828.919698] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8828.939124] alg: No test for adler32 (adler32-zlib) [ 8829.950739] Lustre: Lustre: Build Version: 2.17.57_110_g5f5d390 [ 8830.169525] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 8831.887207] Key type lgssc registered [ 8832.954728] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8883.043443] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8883.058673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8884.315223] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8884.349222] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8884.433140] Lustre: lustre-MDT0000: new disk, initializing [ 8884.500640] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8884.524817] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8889.035481] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 8900.205366] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8900.432064] Lustre: 297124:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8900.527143] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8900.531270] Lustre: Skipped 1 previous similar message [ 8900.620302] Lustre: lustre-MDT0001: new disk, initializing [ 8900.758722] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8900.810572] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8900.836149] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8905.662644] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 8915.601265] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8920.223057] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8925.602106] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8925.966622] Lustre: lustre-OST2710: new disk, initializing [ 8925.970422] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8925.975693] Lustre: 299094:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8926.069208] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8931.788179] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 8934.459108] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8934.465680] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8934.583298] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8939.697394] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8945.166645] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8945.327550] Lustre: lustre-OST03e8: new disk, initializing [ 8945.330841] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8945.336805] Lustre: 300158:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8945.395015] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8950.179711] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 8951.876032] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8951.886170] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8951.920544] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8956.493993] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8963.159206] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8963.338086] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8966.947721] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8967.123224] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8971.566153] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8971.761975] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8976.658696] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8976.874811] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8982.501110] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8982.525085] Lustre: Skipped 1 previous similar message [ 8982.528906] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8982.536666] Lustre: Skipped 2 previous similar messages [ 8983.467668] Lustre: server umount lustre-MDT0000 complete [ 8986.777485] LustreError: 297117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789790094 with bad export cookie 18442155037614519493 [ 8986.782079] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8986.794407] LustreError: 297117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8987.112121] Lustre: server umount lustre-MDT0001 complete [ 9000.188981] Lustre: server umount lustre-OST2710 complete [ 9003.103212] Lustre: 294919:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789790095/real 1789790095] req@0000000071233096 x1876730774092544/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789790111 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9003.132126] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9003.147741] Lustre: Skipped 2 previous similar messages [ 9003.464459] Lustre: server umount lustre-OST03e8 complete [ 9018.567588] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 9026.782314] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 9058.638904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9058.830460] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9058.854105] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9058.916732] Lustre: lustre-MDT0000: new disk, initializing [ 9058.987153] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9059.007879] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9062.878482] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9073.522135] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9073.644559] Lustre: 305403:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9073.708116] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9073.714088] Lustre: Skipped 1 previous similar message [ 9073.782823] Lustre: lustre-MDT0001: new disk, initializing [ 9073.896537] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9073.947256] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9073.955142] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9077.938124] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9087.690588] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9093.118879] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9100.385668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9100.749777] Lustre: lustre-OST0000: new disk, initializing [ 9100.756691] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9100.780502] Lustre: 307376:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9100.917991] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9107.380372] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9110.562473] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9110.573819] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9110.722142] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9116.881889] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9122.092789] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9122.313588] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9126.850276] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9127.084168] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9131.493028] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9131.494261] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9131.499286] LustreError: Skipped 1 previous similar message [ 9131.500430] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9131.520993] Lustre: Skipped 1 previous similar message [ 9136.617676] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9136.628299] Lustre: Skipped 2 previous similar messages [ 9141.728759] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9141.738369] Lustre: Skipped 1 previous similar message [ 9143.263169] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9143.441304] Lustre: server umount lustre-OST0000 complete [ 9146.848433] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9146.860679] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9146.866929] Lustre: Skipped 1 previous similar message [ 9146.877229] Lustre: Skipped 1 previous similar message [ 9157.088986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9157.101417] Lustre: Skipped 3 previous similar messages [ 9161.183217] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9161.428588] Lustre: server umount lustre-MDT0000 complete [ 9164.825684] LustreError: 306201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789790272 with bad export cookie 18442155037614521173 [ 9164.831090] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9164.833814] LustreError: 306201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9165.124445] Lustre: server umount lustre-MDT0001 complete [ 9173.071894] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 23:58:00 (1789790280) [ 9174.434829] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9175.979305] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 23:58:03 (1789790283) [ 9179.441374] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 9203.490980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9203.699826] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9203.721671] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9203.852295] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9207.248535] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9214.784363] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9215.007697] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9215.066877] Lustre: 311233:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9221.707636] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9244.599308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9244.922804] Lustre: Denying initial registration attempt from nid 192.168.203.145@tcp, specified as failover [ 9244.938436] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9244.947768] LustreError: 312686:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -99 [ 9244.963591] LustreError: 312686:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 9244.970560] LustreError: 312686:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9245.080652] Lustre: server umount lustre-OST0000 complete [ 9245.087671] LustreError: 312686:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9254.881634] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9254.897224] Lustre: Skipped 1 previous similar message [ 9254.898418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9254.907697] Lustre: Skipped 1 previous similar message [ 9257.656718] Lustre: server umount lustre-MDT0000 complete [ 9260.001540] LustreError: 311243:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9260.009905] LustreError: 311243:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 9260.727607] LustreError: 311225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789790368 with bad export cookie 18442155037614522055 [ 9260.735934] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9260.736824] LustreError: 311225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9260.972057] Lustre: server umount lustre-MDT0001 complete [ 9270.782157] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 9273.076545] Key type lgssc unregistered [ 9273.326863] LNet: 313806:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9273.343181] LNetError: 313806:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9273.358087] LNet: Removed LNI 192.168.203.145@tcp [ 9274.069581] Key type .llcrypt unregistered [ 9274.074447] Key type ._llcrypt unregistered [ 9284.484345] Key type ._llcrypt registered [ 9284.486086] Key type .llcrypt registered [ 9284.581834] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 9285.496453] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9285.571610] alg: No test for adler32 (adler32-zlib) [ 9286.590323] Lustre: Lustre: Build Version: 2.17.57_110_g5f5d390 [ 9286.834777] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 9288.479169] Key type lgssc registered [ 9289.547606] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9336.236223] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 9345.501901] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9345.519698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9346.850734] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9346.889522] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9347.175238] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9351.533798] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9360.972175] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9361.094746] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9361.136749] Lustre: 316568:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9361.404905] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9366.287958] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9374.138333] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9379.345783] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9399.468899] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9399.750067] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9400.083655] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9407.823121] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9418.130455] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9422.313662] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9422.323249] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9422.330309] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9422.337374] Lustre: Skipped 1 previous similar message [ 9426.239250] Lustre: server umount lustre-OST0000 complete [ 9432.545702] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9432.572085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9432.577583] Lustre: Skipped 2 previous similar messages [ 9437.666675] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9437.681711] Lustre: Skipped 1 previous similar message [ 9442.785106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9442.790292] Lustre: Skipped 1 previous similar message [ 9444.319135] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9444.451865] Lustre: server umount lustre-MDT0000 complete [ 9448.347810] LustreError: 316560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789790556 with bad export cookie 8746190004909822837 [ 9448.359441] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9448.364650] LustreError: 316560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9448.687422] Lustre: server umount lustre-MDT0001 complete [ 9457.921227] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 9460.456245] Key type lgssc unregistered [ 9460.688898] LNet: 320158:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9460.700852] LNetError: 320158:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9460.731015] LNet: Removed LNI 192.168.203.145@tcp [ 9461.427825] Key type .llcrypt unregistered [ 9461.431287] Key type ._llcrypt unregistered [ 9485.586772] Key type ._llcrypt registered [ 9485.591149] Key type .llcrypt registered [ 9485.705668] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 9487.321719] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9487.417131] alg: No test for adler32 (adler32-zlib) [ 9488.373636] Lustre: Lustre: Build Version: 2.17.57_110_g5f5d390 [ 9488.549876] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 9490.319193] Key type lgssc registered [ 9491.275704] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9498.951119] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9498.967751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9500.442235] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9505.518975] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9510.883985] LustreError: 321583:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9510.913469] LustreError: 321583:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9514.516156] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9514.961610] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9520.318070] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9529.443557] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9535.847956] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9544.113133] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9544.633473] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9551.462466] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9558.784192] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9570.276798] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9570.278938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9570.285257] Lustre: Skipped 2 previous similar messages [ 9570.309813] Lustre: Skipped 2 previous similar messages [ 9571.784996] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.45@tcp (stopping) [ 9573.885561] Lustre: server umount lustre-MDT0000 complete [ 9575.394650] LustreError: 322311:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9575.411397] LustreError: 322311:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9576.870276] LustreError: 322545:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9577.305190] LustreError: 321566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789790685 with bad export cookie 6886588800491834391 [ 9577.326581] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9580.521707] LustreError: 321579:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9580.527016] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9580.569776] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9583.518274] Lustre: server umount lustre-MDT0001 complete [ 9589.412671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9598.718744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9602.463874] LustreError: 323416:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9602.485219] LustreError: 323416:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9602.590697] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9602.686763] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9608.144371] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9612.780744] LustreError: 323417:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9612.783348] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9612.809084] LustreError: 323417:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 9616.686512] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9616.935700] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9620.827152] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9622.003295] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9628.221738] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9629.687191] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9634.670489] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9636.157165] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9642.468610] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9642.470430] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9642.483736] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9642.499090] Lustre: Skipped 1 previous similar message [ 9642.514310] Lustre: Skipped 3 previous similar messages [ 9646.771386] Lustre: server umount lustre-OST0000 complete [ 9652.718289] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9652.727312] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9652.759293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9652.771512] Lustre: Skipped 1 previous similar message [ 9657.135322] Lustre: server umount lustre-MDT0000 complete [ 9657.828077] LustreError: 327101:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9657.841591] LustreError: 327101:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9661.099738] LustreError: 321565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789790769 with bad export cookie 6886588800491841965 [ 9661.103518] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9661.129520] LustreError: 321565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 9661.502207] Lustre: server umount lustre-MDT0001 complete [ 9672.347546] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 9675.012191] Key type lgssc unregistered [ 9675.308450] LNet: 327851:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9675.319713] LNetError: 327851:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9675.365944] LNet: Removed LNI 192.168.203.145@tcp [ 9676.327087] Key type .llcrypt unregistered [ 9676.329979] Key type ._llcrypt unregistered [ 9701.593041] Key type ._llcrypt registered [ 9701.595839] Key type .llcrypt registered [ 9701.679626] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 9702.607952] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9702.669655] alg: No test for adler32 (adler32-zlib) [ 9703.713929] Lustre: Lustre: Build Version: 2.17.57_110_g5f5d390 [ 9703.900407] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 9705.552000] Key type lgssc registered [ 9706.774451] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9714.804576] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9714.824808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9716.275083] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9720.315971] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9726.452709] LustreError: 329268:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9726.479806] LustreError: 329268:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9728.700362] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9729.140174] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9733.072074] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9740.253353] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9746.121482] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9749.476933] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9749.486520] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9749.495599] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9752.035561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9752.044669] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9754.146321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9754.158061] Lustre: Skipped 2 previous similar messages [ 9759.200381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9759.207312] Lustre: Skipped 1 previous similar message [ 9763.295155] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9763.595108] Lustre: server umount lustre-MDT0000 complete [ 9764.327096] LustreError: 329268:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9764.343866] LustreError: 329268:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9767.117709] LustreError: 329249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789790875 with bad export cookie 8148278786896560392 [ 9767.134050] LustreError: 329249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9767.134873] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9767.484708] Lustre: server umount lustre-MDT0001 complete [ 9783.173362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9783.516655] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9783.556493] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9783.766088] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9788.353505] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9796.566269] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9796.680127] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9796.701403] Lustre: 331856:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9797.033846] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9801.342814] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9808.767032] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9815.059754] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9824.813936] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9825.036219] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9825.045647] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9825.048682] LustreError: 333739:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -2 [ 9825.058908] LustreError: 333739:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 9825.064778] LustreError: 333739:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9825.127496] Lustre: server umount lustre-OST0000 complete [ 9825.131694] LustreError: 333739:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9834.859931] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9835.044867] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9835.258674] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9842.318319] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9843.924386] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9851.074233] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9859.514105] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9859.628805] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9859.721517] Lustre: lustre-OST0001: new disk, initializing [ 9859.725549] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9859.732342] Lustre: 335283:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9859.833408] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9865.996668] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9868.316825] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9868.326034] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9868.440860] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9872.680767] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9878.510468] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9878.511975] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9878.522197] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9878.532780] Lustre: Skipped 1 previous similar message [ 9878.553358] Lustre: Skipped 1 previous similar message [ 9888.747237] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9888.754917] Lustre: Skipped 2 previous similar messages [ 9890.271485] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9890.369730] Lustre: server umount lustre-OST0001 complete [ 9893.871907] LustreError: 334204:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9893.889441] LustreError: 334204:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9894.880427] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9894.892344] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9898.987077] LustreError: 334205:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9899.004751] LustreError: 334205:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9900.768912] Lustre: server umount lustre-OST0000 complete [ 9904.616870] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9904.624262] LustreError: Skipped 1 previous similar message [ 9904.627833] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9904.638223] Lustre: Skipped 1 previous similar message [ 9909.217523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9909.228204] Lustre: Skipped 6 previous similar messages [ 9910.684652] Lustre: server umount lustre-MDT0000 complete [ 9913.737970] LustreError: 331848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789791021 with bad export cookie 8148278786896560945 [ 9913.742374] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9913.763853] LustreError: 331848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9914.171625] Lustre: server umount lustre-MDT0001 complete [ 9925.895361] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 9928.357396] Key type lgssc unregistered [ 9928.696969] LNet: 337240:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9928.707437] LNetError: 337240:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9928.725153] LNet: Removed LNI 192.168.203.145@tcp [ 9929.436138] Key type .llcrypt unregistered [ 9929.438373] Key type ._llcrypt unregistered [10001.709425] Key type ._llcrypt registered [10001.712234] Key type .llcrypt registered [10001.922411] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [10018.553162] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [10019.797359] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10019.851957] alg: No test for adler32 (adler32-zlib) [10020.965323] Lustre: Lustre: Build Version: 2.17.57_110_g5f5d390 [10021.236493] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [10022.911290] Key type lgssc registered [10024.098879] Lustre: Echo OBD driver; http://www.lustre.org/ [10059.246106] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10059.320924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10060.621673] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10060.702215] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10060.815583] Lustre: lustre-MDT0000: new disk, initializing [10060.925570] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10060.964679] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10065.217922] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [10075.937815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10076.091594] Lustre: 342173:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10076.166969] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10076.171148] Lustre: Skipped 1 previous similar message [10076.288178] Lustre: lustre-MDT0001: new disk, initializing [10076.392145] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10076.474249] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10076.493438] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10080.924875] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [10089.800948] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10095.877414] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10103.161035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10103.447356] Lustre: lustre-OST0000: new disk, initializing [10103.454817] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10103.468450] Lustre: 344144:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10103.588690] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10109.874964] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [10111.549576] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10111.558811] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10111.674386] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10119.545425] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10124.346939] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10124.571230] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10129.540289] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10129.751259] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10132.451112] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10132.459240] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10132.462154] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10132.474847] Lustre: Skipped 1 previous similar message [10137.526744] Lustre: server umount lustre-OST0000 complete [10142.690642] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10142.706570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10142.718966] Lustre: Skipped 1 previous similar message [10142.724159] Lustre: Skipped 1 previous similar message [10147.328705] Lustre: server umount lustre-MDT0000 complete [10150.788361] LustreError: 344143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789791258 with bad export cookie 9897475394285869099 [10150.795827] LustreError: 344143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10150.797987] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10151.086828] Lustre: server umount lustre-MDT0001 complete [10159.217681] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 00:14:26 (1789791266) [10167.038625] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 00:14:34 (1789791274) [10174.842306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10175.327063] LustreError: 346900:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10175.354564] LustreError: 346900:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10175.403443] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10179.716078] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [10180.576235] LustreError: 346901:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10185.696546] LustreError: 346900:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10187.793459] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10188.053437] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10192.362355] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [10199.937251] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10205.879891] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10215.652945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10215.930954] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10222.754146] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [10231.977506] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10240.483340] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10242.398859] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10244.337329] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10246.389717] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10248.707970] Lustre: Failing over lustre-MDT0000 [10248.718490] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10248.747609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10248.756946] Lustre: Skipped 1 previous similar message [10249.112601] Lustre: server umount lustre-MDT0000 complete [10249.699427] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10249.708659] LustreError: Skipped 1 previous similar message [10249.714827] LustreError: 346901:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10253.794750] LustreError: 346897:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10253.800061] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10253.829576] LustreError: 346897:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10253.861619] Lustre: Skipped 1 previous similar message [10264.038154] LustreError: 346901:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10264.066957] LustreError: 346901:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [10267.639594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10267.764463] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10268.062311] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10268.116432] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10272.170239] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10272.457591] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [10273.265575] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10273.313031] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [10273.356320] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10281.315841] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10282.731102] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10286.501765] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10288.367874] Lustre: Failing over lustre-MDT0000 [10288.608414] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10288.621464] Lustre: Skipped 1 previous similar message [10288.628639] LustreError: 348523:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10288.638000] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10288.643907] LustreError: 348523:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10288.778282] Lustre: server umount lustre-MDT0000 complete [10307.795137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10307.993409] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10308.352488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10308.409457] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10312.611485] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10312.726515] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [10313.709309] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10313.719021] Lustre: Skipped 2 previous similar messages [10313.745987] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [10313.816628] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10321.973481] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10323.586233] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10329.060365] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10329.066712] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10329.067868] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10329.085481] Lustre: Skipped 3 previous similar messages [10329.092067] Lustre: Skipped 2 previous similar messages [10334.177194] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10334.183390] Lustre: Skipped 2 previous similar messages [10334.845887] Lustre: server umount lustre-MDT0000 complete [10337.911956] LustreError: 346880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789791445 with bad export cookie 9897475394285871815 [10337.916712] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10337.929574] LustreError: 346880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10338.194818] Lustre: server umount lustre-MDT0001 complete [10351.287080] Lustre: server umount lustre-OST0000 complete [10366.218475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10366.811258] LustreError: 353245:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10366.838027] LustreError: 353245:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [10371.890943] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [10380.788347] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10381.097637] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10381.101686] Lustre: Skipped 1 previous similar message [10385.456326] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [10392.788503] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10398.904444] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10405.834665] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10411.311317] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [10411.313053] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10418.537197] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10425.368696] Lustre: server umount lustre-OST0000 complete [10431.969722] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10431.972955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10432.001708] Lustre: Skipped 1 previous similar message [10432.014955] Lustre: Skipped 1 previous similar message [10439.310262] Lustre: server umount lustre-MDT0000 complete [10442.209555] Lustre: 340395:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789791534/real 1789791534] req@0000000083799190 x1876732023020032/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789791550 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10442.220546] LustreError: 354833:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10442.256159] LustreError: 354833:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10443.295800] Lustre: 340395:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789791535/real 1789791535] req@00000000e9684581 x1876732023020544/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789791551 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10443.554246] LustreError: 353226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789791551 with bad export cookie 9897475394285872536 [10443.557689] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10443.564940] LustreError: 353226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10444.037968] Lustre: server umount lustre-MDT0001 complete [10457.153915] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [10459.977387] Key type lgssc unregistered [10460.284788] LNet: 356775:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10460.307730] LNetError: 356775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10460.328133] LNet: Removed LNI 192.168.203.145@tcp [10461.242865] Key type .llcrypt unregistered [10461.246792] Key type ._llcrypt unregistered [10487.655915] Key type ._llcrypt registered [10487.657721] Key type .llcrypt registered [10487.753615] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [10488.903746] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10488.920531] alg: No test for adler32 (adler32-zlib) [10489.951237] Lustre: Lustre: Build Version: 2.17.57_110_g5f5d390 [10490.167601] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [10491.815236] Key type lgssc registered [10492.795988] Lustre: Echo OBD driver; http://www.lustre.org/ [10500.076939] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10500.104988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10501.586840] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10506.116768] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [10511.843446] LustreError: 358290:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10511.854606] LustreError: 358290:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10513.568321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10513.864709] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10517.526949] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [10524.101336] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10529.629555] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10535.380158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10535.689629] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10541.290723] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [10546.996783] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10548.606931] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10559.973868] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10559.979614] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10559.990872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10562.528801] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10562.531988] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10562.534607] Lustre: Skipped 1 previous similar message [10562.553087] Lustre: Skipped 1 previous similar message [10565.068800] Lustre: server umount lustre-MDT0000 complete [10567.653080] LustreError: 358286:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10567.667321] LustreError: 358286:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10568.569682] LustreError: 359025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789791676 with bad export cookie 3906639995152067079 [10568.573931] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10568.583813] LustreError: 359025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10568.812376] Lustre: server umount lustre-MDT0001 complete [10582.433791] Lustre: server umount lustre-OST0000 complete [10595.221353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10595.538965] LustreError: 361999:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10595.579916] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10600.936996] LustreError: 362000:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10600.985298] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [10605.026640] LustreError: 361999:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10609.154203] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10609.458679] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10613.295217] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [10620.009572] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10626.224577] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10632.784276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10633.142248] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10639.259429] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [10641.335260] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10645.828702] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10660.840674] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10660.846561] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10660.873095] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10663.564778] Lustre: server umount lustre-MDT0000 complete [10666.985666] LustreError: 362000:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10667.003192] LustreError: 362000:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10667.013603] LustreError: 361979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789791775 with bad export cookie 3906639995152068192 [10667.016073] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10667.020911] LustreError: 361979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10667.425149] Lustre: server umount lustre-MDT0001 complete [10681.483062] Lustre: server umount lustre-OST0000 complete [10695.697316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10696.146280] LustreError: 365910:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10696.205697] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10699.753139] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [10708.386606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10714.538035] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [10721.619670] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10727.406604] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10733.699228] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10734.014191] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10734.021927] Lustre: Skipped 1 previous similar message [10739.652614] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [10741.169893] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10745.983951] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10751.460674] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10751.466393] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10751.490146] Lustre: Skipped 2 previous similar messages [10751.497199] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10751.504230] Lustre: Skipped 3 previous similar messages [10756.272489] Lustre: server umount lustre-OST0000 complete [10760.161548] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10760.176191] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10760.202196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10760.211969] Lustre: Skipped 2 previous similar messages [10766.376672] Lustre: server umount lustre-MDT0000 complete [10768.869403] LustreError: 365911:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10768.888723] LustreError: 365911:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10770.435583] LustreError: 365890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789791878 with bad export cookie 3906639995152069221 [10770.438539] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10770.466513] LustreError: 365890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10770.771526] Lustre: server umount lustre-MDT0001 complete [10784.156173] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [10787.527938] Key type lgssc unregistered [10787.949251] LNet: 369442:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10787.952335] LNetError: 369442:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10787.975165] LNet: Removed LNI 192.168.203.145@tcp [10788.967675] Key type .llcrypt unregistered [10788.971573] Key type ._llcrypt unregistered [10814.589242] Key type ._llcrypt registered [10814.590398] Key type .llcrypt registered [10814.658341] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [10815.477287] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10815.551250] alg: No test for adler32 (adler32-zlib) [10816.661150] Lustre: Lustre: Build Version: 2.17.57_110_g5f5d390 [10816.892691] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [10818.615788] Key type lgssc registered [10819.632338] Lustre: Echo OBD driver; http://www.lustre.org/ [10827.853163] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10827.888529] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10827.910411] LustreError: 370880:0:(osd_handler.c:8547:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10827.947771] LustreError: 370880:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10827.959136] LustreError: 370880:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10827.964699] LustreError: 370880:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10827.971240] LustreError: 370880:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10835.975288] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10835.982653] LustreError: 371277:0:(osd_handler.c:8547:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10836.012321] LustreError: 371277:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10836.031890] LustreError: 371277:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10836.039570] LustreError: 371277:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10836.059738] LustreError: 371277:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10855.824601] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [10858.602102] Key type lgssc unregistered [10858.941807] LNet: 372280:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10858.950122] LNetError: 372280:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10858.965100] LNet: Removed LNI 192.168.203.145@tcp [10859.659544] Key type .llcrypt unregistered [10859.663895] Key type ._llcrypt unregistered [10877.909767] Key type ._llcrypt registered [10877.912293] Key type .llcrypt registered [10878.010398] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [10892.725520] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [10893.473443] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10893.515232] alg: No test for adler32 (adler32-zlib) [10894.586284] Lustre: Lustre: Build Version: 2.17.57_110_g5f5d390 [10894.798065] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [10896.455218] Key type lgssc registered [10897.585320] Lustre: Echo OBD driver; http://www.lustre.org/ [10931.206710] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10931.256628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10932.574730] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10932.621965] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10932.742679] Lustre: lustre-MDT0000: new disk, initializing [10932.912252] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10932.942295] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10937.758418] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [10949.047617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10949.173991] Lustre: 375435:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10949.283609] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10949.288942] Lustre: Skipped 1 previous similar message [10949.341370] Lustre: lustre-MDT0001: new disk, initializing [10949.428147] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10949.452389] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10949.461523] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10953.482905] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [10962.705049] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10968.806943] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10976.567655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10977.044635] Lustre: lustre-OST0000: new disk, initializing [10977.049176] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10977.054923] Lustre: 377406:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10977.203269] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10984.028418] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [10986.033714] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10986.049132] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10986.141814] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10993.964478] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10998.349784] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10998.558807] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11002.644473] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11002.888907] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11006.954269] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11006.965321] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11006.978560] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11006.986671] Lustre: Skipped 1 previous similar message [11010.866706] Lustre: server umount lustre-OST0000 complete [11016.160704] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11016.167295] LustreError: Skipped 1 previous similar message [11016.175744] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11016.187903] Lustre: Skipped 1 previous similar message [11016.197946] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11017.190609] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11017.205668] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11017.217467] Lustre: Skipped 1 previous similar message [11020.220203] Lustre: server umount lustre-MDT0000 complete [11023.635197] LustreError: 377405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789792131 with bad export cookie 2945358521832690588 [11023.639054] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11023.650111] LustreError: 377405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11023.939670] Lustre: server umount lustre-MDT0001 complete [11032.713746] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11035.276313] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 00:29:02 (1789792142) [11042.491600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11042.960321] LustreError: 379866:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11042.981330] LustreError: 379866:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11043.033735] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11048.418762] LustreError: 379867:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11048.444864] LustreError: 379867:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11048.855679] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [11053.538573] LustreError: 379866:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11058.664778] LustreError: 379867:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11058.962633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11059.403425] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11063.665391] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [11070.835446] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11077.128562] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11084.889433] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11085.338431] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11091.843675] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [11099.254774] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11106.373181] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11106.559672] Lustre: lustre-OST0001: new disk, initializing [11106.564723] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11106.575824] Lustre: 382700:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11106.666823] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11113.263632] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [11114.104067] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11114.117502] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11114.231740] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11123.153652] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11129.831547] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11129.832498] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11129.843166] LustreError: Skipped 1 previous similar message [11129.850033] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11129.867436] Lustre: Skipped 1 previous similar message [11129.882707] Lustre: Skipped 1 previous similar message [11133.021052] Lustre: server umount lustre-OST0001 complete [11134.955790] LustreError: 381688:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11134.973178] LustreError: 381688:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11145.192712] LustreError: 383116:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11145.222464] LustreError: 383116:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [11150.307430] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11150.313286] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11150.325756] Lustre: Skipped 1 previous similar message [11150.340081] Lustre: Skipped 2 previous similar messages [11156.242581] Lustre: server umount lustre-OST0000 complete [11160.544937] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11160.568752] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11160.572848] Lustre: Skipped 2 previous similar messages [11174.367319] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11174.545937] Lustre: server umount lustre-MDT0000 complete [11175.904224] LustreError: 379866:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11175.921455] LustreError: 379866:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11178.618186] LustreError: 379846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789792286 with bad export cookie 2945358521832691470 [11178.620274] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11178.627336] LustreError: 379846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11185.117972] Lustre: server umount lustre-MDT0001 complete [11197.440642] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [11200.444943] Key type lgssc unregistered [11200.706139] LNet: 384691:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11200.711242] LNetError: 384691:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11201.770916] LNet: Removed LNI 192.168.203.145@tcp [11202.575298] Key type .llcrypt unregistered [11202.580642] Key type ._llcrypt unregistered [11252.122918] Key type ._llcrypt registered [11252.124962] Key type .llcrypt registered [11252.226881] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [11253.040733] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11253.101255] alg: No test for adler32 (adler32-zlib) [11254.157540] Lustre: Lustre: Build Version: 2.17.57_110_g5f5d390 [11254.361721] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [11256.050000] Key type lgssc registered [11257.169915] Lustre: Echo OBD driver; http://www.lustre.org/ [11264.828503] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11264.848313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11266.190173] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11266.214956] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11266.384752] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11270.740827] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [11278.443408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11278.532040] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11278.583345] Lustre: 387486:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11278.815031] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11283.081191] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [11289.730102] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11294.884431] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11298.785472] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11298.788207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11298.806549] Lustre: Skipped 1 previous similar message [11303.904563] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11303.915710] Lustre: Skipped 2 previous similar messages [11309.025455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11309.031989] Lustre: Skipped 1 previous similar message [11311.074178] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11311.173155] Lustre: server umount lustre-MDT0000 complete [11314.792883] LustreError: 387481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789792422 with bad export cookie 17528638449107532866 [11314.802078] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11314.809062] LustreError: 387481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11315.182443] Lustre: server umount lustre-MDT0001 complete [11324.870504] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 00:33:52 (1789792432) [11331.747614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11332.257889] LustreError: 389999:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11332.273360] LustreError: 389999:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11332.318861] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11336.666238] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [11337.696729] LustreError: 390000:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11342.820932] LustreError: 389999:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11344.451706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11344.801272] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11348.891537] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [11356.138267] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11361.995960] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11368.510846] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11368.705788] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11368.960539] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11374.706088] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [11382.171349] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11390.115891] Lustre: Permanently deactivating lustre-OST0000 [11390.128369] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11396.146424] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11397.114363] LustreError: 392938:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11397.124921] LustreError: 392938:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11397.131923] LustreError: 392938:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11399.771296] LustreError: 392986:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11399.795657] LustreError: 392986:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11407.640547] Lustre: server umount lustre-OST0000 complete [11409.910761] LustreError: 393236:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11409.915564] LustreError: 393236:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11412.449331] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11412.450722] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11412.466193] Lustre: Skipped 1 previous similar message [11417.574818] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11417.583280] Lustre: Skipped 2 previous similar messages [11421.738951] Lustre: server umount lustre-MDT0000 complete [11422.697537] LustreError: 389996:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11422.724747] LustreError: 389996:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11425.272612] LustreError: 389979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789792533 with bad export cookie 17528638449107533454 [11425.279287] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11425.286589] LustreError: 389979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11435.606285] Lustre: server umount lustre-MDT0001 complete [11442.916651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11447.545742] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [11450.624313] LustreError: 394394:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11457.665190] Lustre: 394734:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.145@tcp [11462.182324] Lustre: server umount MGS complete [11482.571367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11483.436578] LustreError: 395460:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11483.499632] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11483.509545] Lustre: Skipped 1 previous similar message [11483.690597] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11489.455744] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [11493.858697] LustreError: 395461:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11493.879084] LustreError: 395461:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11497.763710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11498.127517] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11502.536985] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [11509.721606] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11515.780211] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11518.114779] Lustre: Permanently reactivating lustre-OST0000 [11518.135590] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11518.144471] Lustre: Skipped 2 previous similar messages [11548.644349] LustreError: 397290:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11554.991505] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11555.247906] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11555.252571] Lustre: Skipped 1 previous similar message [11560.525366] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [11567.352407] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11573.729566] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11573.730032] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11573.756531] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11573.782951] Lustre: Skipped 1 previous similar message [11578.888835] Lustre: server umount lustre-OST0000 complete [11583.968671] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11583.982923] Lustre: Skipped 2 previous similar messages [11593.186854] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11593.193327] Lustre: Skipped 5 previous similar messages [11597.279348] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11597.440399] Lustre: server umount lustre-MDT0000 complete [11598.306422] LustreError: 395461:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11598.335138] LustreError: 395461:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11601.290436] LustreError: 395440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789792709 with bad export cookie 17528638449107534504 [11601.298031] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11601.300721] LustreError: 395440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11601.681533] Lustre: server umount lustre-MDT0001 complete [11612.143177] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [11614.546518] Key type lgssc unregistered [11614.809613] LNet: 399921:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11614.823831] LNetError: 399921:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11614.845280] LNet: Removed LNI 192.168.203.145@tcp [11615.594485] Key type .llcrypt unregistered [11615.597218] Key type ._llcrypt unregistered [11635.158794] Key type ._llcrypt registered [11635.160510] Key type .llcrypt registered [11635.268736] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [11652.881887] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [11654.101924] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11654.233467] alg: No test for adler32 (adler32-zlib) [11655.405918] Lustre: Lustre: Build Version: 2.17.57_110_g5f5d390 [11655.752723] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [11657.447182] Key type lgssc registered [11658.551722] Lustre: Echo OBD driver; http://www.lustre.org/ [11695.992802] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11696.035557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11697.241141] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11697.291338] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11697.397979] Lustre: lustre-MDT0000: new disk, initializing [11697.486118] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11697.505374] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11702.369192] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [11713.123148] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11713.266492] Lustre: 403076:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11713.316771] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11713.323263] Lustre: Skipped 1 previous similar message [11713.380836] Lustre: lustre-MDT0001: new disk, initializing [11713.481370] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11713.533683] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11713.543679] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11717.918845] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [11727.725595] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11733.841814] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11740.303750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11740.521325] Lustre: lustre-OST0000: new disk, initializing [11740.525820] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11740.532893] Lustre: 405044:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11740.612286] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11746.478161] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [11747.408258] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11747.421176] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11747.548932] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11756.225653] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11761.047238] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11761.377418] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11766.073249] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11766.314491] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11768.293185] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11768.293913] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11768.312526] LustreError: Skipped 1 previous similar message [11768.338693] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11768.345919] Lustre: Skipped 1 previous similar message [11773.411732] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11773.421070] Lustre: Skipped 1 previous similar message [11774.463856] Lustre: server umount lustre-OST0000 complete [11778.529875] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11778.540035] Lustre: Skipped 1 previous similar message [11778.543766] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11783.650320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11783.666293] Lustre: Skipped 2 previous similar messages [11784.783091] Lustre: server umount lustre-MDT0000 complete [11788.977531] LustreError: 403871:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789792897 with bad export cookie 5267689600416504010 [11788.992991] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11788.994708] LustreError: 403871:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11789.422549] Lustre: server umount lustre-MDT0001 complete [11798.557149] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11800.709803] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 00:41:47 (1789792907) [11808.032156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11808.474383] LustreError: 407503:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11808.489555] LustreError: 407503:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11808.535684] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11813.491292] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [11813.857544] LustreError: 407504:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11818.978479] LustreError: 407503:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11821.998330] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11822.427713] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11826.929288] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [11835.031733] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11841.648355] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11848.565604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11848.991569] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11855.424201] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [11863.683427] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11865.967821] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11868.545258] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11874.785311] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11874.792492] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11874.817450] Lustre: Skipped 1 previous similar message [11874.821206] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11877.860373] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11877.867519] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11880.290894] Lustre: server umount lustre-OST0000 complete [11888.099019] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11888.106854] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11888.110036] Lustre: Skipped 3 previous similar messages [11890.696803] Lustre: server umount lustre-MDT0000 complete [11893.216400] LustreError: 407503:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11893.237239] LustreError: 407503:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11894.347147] LustreError: 407484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789793002 with bad export cookie 5267689600416504892 [11894.356902] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11894.362039] LustreError: 407484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11894.824624] Lustre: server umount lustre-MDT0001 complete [11909.174243] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [11912.906778] Key type lgssc unregistered [11913.384848] LNet: 411231:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11913.411098] LNetError: 411231:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11913.450653] LNet: Removed LNI 192.168.203.145@tcp [11915.087046] Key type .llcrypt unregistered [11915.091709] Key type ._llcrypt unregistered [11941.312680] Key type ._llcrypt registered [11941.315239] Key type .llcrypt registered [11941.495738] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [11941.822725] Key type .llcrypt unregistered [11941.826524] Key type ._llcrypt unregistered [11960.846655] Key type ._llcrypt registered [11960.849064] Key type .llcrypt registered [11960.970695] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [11962.341550] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11962.410448] alg: No test for adler32 (adler32-zlib) [11963.570155] Lustre: Lustre: Build Version: 2.17.57_110_g5f5d390 [11963.955546] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [11965.687354] Key type lgssc registered [11966.875555] Lustre: Echo OBD driver; http://www.lustre.org/ [11975.642228] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11975.664185] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11977.246970] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11981.761175] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [11987.424916] LustreError: 413439:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11987.442227] LustreError: 413439:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11989.982557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11990.301475] LustreError: 413440:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11990.328229] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11995.620026] LustreError: 413435:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11995.643994] LustreError: 413435:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11995.702328] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [12000.742816] LustreError: 413439:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12000.777972] LustreError: 413439:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12002.805514] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12004.127548] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12007.907821] LustreError: 413440:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12007.953245] LustreError: 413440:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12012.829870] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12013.470353] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12019.330314] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [12028.200435] Lustre: server umount lustre-OST0000 complete [12034.017222] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12034.022327] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12034.024100] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12034.050226] Lustre: Skipped 1 previous similar message [12039.136033] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12039.143879] Lustre: Skipped 2 previous similar messages [12044.259791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12044.268868] Lustre: Skipped 1 previous similar message [12044.964322] Lustre: 412748:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789793136/real 1789793136] req@00000000483162fe x1876734059243264/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789793152 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12045.000843] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12046.304169] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12046.517961] Lustre: server umount lustre-MDT0000 complete [12050.128823] LustreError: 413420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789793158 with bad export cookie 16430048208754014918 [12050.133341] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12050.138774] LustreError: 413420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12050.425287] Lustre: server umount lustre-MDT0001 complete [12062.296866] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [12064.764562] Key type lgssc unregistered [12065.041157] LNet: 416585:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12065.055308] LNetError: 416585:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12065.085602] LNet: Removed LNI 192.168.203.145@tcp [12065.924295] Key type .llcrypt unregistered [12065.929283] Key type ._llcrypt unregistered [12091.545797] Key type ._llcrypt registered [12091.553454] Key type .llcrypt registered [12091.663281] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [12093.377381] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12093.408435] alg: No test for adler32 (adler32-zlib) [12094.614373] Lustre: Lustre: Build Version: 2.17.57_110_g5f5d390 [12095.046102] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [12096.823372] Key type lgssc registered [12098.516758] Lustre: Echo OBD driver; http://www.lustre.org/ [12106.493225] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12106.508669] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12122.015506] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12127.135281] LustreError: 417972:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.145@tcp: failed processing log, type 4: rc = -110 [12153.760792] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12160.587384] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [12168.657979] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12174.918608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12174.988955] LustreError: 417307:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000003d58d91e x1876734197631232/t0(0) o250->MGC192.168.203.145@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12175.308950] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12175.377083] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12179.496162] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [12180.456905] LustreError: 417998:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12180.486315] LustreError: 417998:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12185.572973] LustreError: 417997:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12187.283636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12187.686794] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12191.900911] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [12199.782630] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12205.512714] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12213.216775] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12213.225905] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12213.240944] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12214.244834] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12214.249535] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12214.262617] Lustre: Skipped 1 previous similar message [12215.252738] Lustre: server umount lustre-OST0000 complete [12219.364278] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12219.369111] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12219.394601] Lustre: Skipped 1 previous similar message [12219.414736] Lustre: Skipped 1 previous similar message [12220.806601] Lustre: server umount lustre-MDT0000 complete [12224.482573] LustreError: 419014:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12224.506957] LustreError: 419014:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12224.529124] LustreError: 417979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789793332 with bad export cookie 3773425692507959929 [12224.539360] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12224.550212] LustreError: 417979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12224.928248] Lustre: server umount lustre-MDT0001 complete [12236.438129] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [12239.744152] Key type lgssc unregistered [12240.057480] LNet: 421539:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12240.069384] LNetError: 421539:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12240.086375] LNet: Removed LNI 192.168.203.145@tcp [12240.955808] Key type .llcrypt unregistered [12240.958216] Key type ._llcrypt unregistered [12265.747774] Key type ._llcrypt registered [12265.750042] Key type .llcrypt registered [12265.864268] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [12266.994119] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12267.047800] alg: No test for adler32 (adler32-zlib) [12268.062560] Lustre: Lustre: Build Version: 2.17.57_110_g5f5d390 [12268.300794] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [12270.007172] Key type lgssc registered [12271.654746] Lustre: Echo OBD driver; http://www.lustre.org/ [12281.374369] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12281.402676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12283.334985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12288.370300] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [12293.601982] LustreError: 422955:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12293.624289] LustreError: 422955:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12297.192802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12297.704544] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12302.287639] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [12309.748148] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12315.054942] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12321.764062] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12322.192130] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12328.587083] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [12329.389275] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12335.948539] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12343.772393] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.45@tcp (stopping) [12343.780055] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12343.823358] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12344.800447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12344.802241] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12344.812840] Lustre: Skipped 1 previous similar message [12344.843642] Lustre: Skipped 1 previous similar message [12347.053793] Lustre: server umount lustre-MDT0000 complete [12348.833297] LustreError: 422950:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12349.925426] LustreError: 424390:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12349.941889] LustreError: 424390:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12350.545355] LustreError: 422935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.45@tcp arrived at 1789793458 with bad export cookie 12014181253452578665 [12350.563791] LustreError: 422935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12352.946188] LustreError: 422936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789793460 with bad export cookie 12014181253452577769 [12352.956186] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12352.968404] LustreError: 422936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12354.016555] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12354.032709] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12354.054421] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12354.062455] Lustre: Skipped 2 previous similar messages [12355.044850] LustreError: 422956:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12355.077697] LustreError: 422956:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12357.095937] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12357.105447] Lustre: Skipped 1 previous similar message [12359.287670] Lustre: server umount lustre-OST0000 complete [12362.208219] LustreError: 422951:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12362.229840] LustreError: 422951:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12366.041813] Lustre: server umount lustre-MDT0001 complete [12378.292854] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [12381.634522] Key type lgssc unregistered [12381.993437] LNet: 426637:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12382.004192] LNetError: 426637:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12382.025640] LNet: Removed LNI 192.168.203.145@tcp [12383.110550] Key type .llcrypt unregistered [12383.114639] Key type ._llcrypt unregistered [12409.652360] Key type ._llcrypt registered [12409.654350] Key type .llcrypt registered [12409.764703] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [12410.875274] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12411.069857] alg: No test for adler32 (adler32-zlib) [12412.163207] Lustre: Lustre: Build Version: 2.17.57_110_g5f5d390 [12412.417759] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [12414.143923] Key type lgssc registered [12415.444299] Lustre: Echo OBD driver; http://www.lustre.org/ [12424.305628] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12424.327869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12425.984650] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12430.844068] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [12435.430884] LustreError: 428050:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12435.444575] LustreError: 428050:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12439.947315] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12440.457924] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12446.234945] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [12454.200403] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12460.278423] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12467.416545] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12467.845381] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12470.970268] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12474.593958] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [12482.724156] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12489.646373] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.45@tcp (stopping) [12490.733653] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12490.752147] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12490.771811] Lustre: Skipped 1 previous similar message [12493.364902] Lustre: server umount lustre-MDT0001 complete [12494.763843] LustreError: 428046:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12495.843583] LustreError: 428051:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12495.865303] LustreError: 428051:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12500.963763] LustreError: 428779:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12500.965795] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12501.007105] Lustre: Skipped 1 previous similar message [12501.015322] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12506.080500] LustreError: 428050:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12506.086878] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12506.783686] Lustre: server umount lustre-OST0000 complete [12511.356473] Lustre: server umount lustre-MDT0000 complete [12527.991774] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [12531.121146] Key type lgssc unregistered [12531.745943] LNet: 431736:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12531.781834] LNetError: 431736:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12532.840643] LNet: Removed LNI 192.168.203.145@tcp [12534.039055] Key type .llcrypt unregistered [12534.041731] Key type ._llcrypt unregistered [12562.188372] Key type ._llcrypt registered [12562.195182] Key type .llcrypt registered [12562.444437] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [12578.753731] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [12579.888644] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12580.056738] alg: No test for adler32 (adler32-zlib) [12581.176294] Lustre: Lustre: Build Version: 2.17.57_110_g5f5d390 [12581.627806] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [12583.359165] Key type lgssc registered [12584.420448] Lustre: Echo OBD driver; http://www.lustre.org/ [12626.927422] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [12645.271800] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12645.299313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12646.646336] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12646.695070] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12646.830226] Lustre: lustre-MDT0000: new disk, initializing [12646.954647] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12646.982964] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12652.658442] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [12668.379037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12668.620388] Lustre: lustre-OST0000: new disk, initializing [12668.623934] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12668.627820] Lustre: Skipped 1 previous similar message [12668.634141] Lustre: 436769:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12668.707265] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12674.917563] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [12678.239374] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12678.245464] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12678.277526] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12684.769330] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12701.602601] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12701.742634] Lustre: 435752:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12701.819156] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12701.905912] Lustre: lustre-MDT0001: new disk, initializing [12702.041161] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12702.100099] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12702.128836] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12707.294776] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [12718.623561] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12718.844997] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12724.059941] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12724.333606] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12730.850258] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12730.854808] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12730.873466] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12730.893167] Lustre: Skipped 1 previous similar message [12732.906611] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.45@tcp (stopping) [12735.714150] Lustre: server umount lustre-MDT0001 complete [12741.089131] LustreError: 435759:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12741.117198] LustreError: 435759:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [12746.216472] LustreError: 435763:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12746.233793] LustreError: 435763:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12751.330039] LustreError: 435760:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12751.346646] LustreError: 435760:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12754.912185] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12754.916457] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12754.924676] Lustre: Skipped 1 previous similar message [12754.933434] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12756.476260] LustreError: 435763:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12760.382511] Lustre: server umount lustre-OST0000 complete [12761.571899] LustreError: 435764:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12770.958254] Lustre: server umount lustre-MDT0000 complete [12785.019474] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [12787.637245] Key type lgssc unregistered [12788.035977] LNet: 440517:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12788.046887] LNetError: 440517:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12788.073619] LNet: Removed LNI 192.168.203.145@tcp [12788.963657] Key type .llcrypt unregistered [12788.966356] Key type ._llcrypt unregistered [12832.438781] Key type ._llcrypt registered [12832.441166] Key type .llcrypt registered [12832.516398] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [12832.760558] Key type .llcrypt unregistered [12832.763272] Key type ._llcrypt unregistered [12848.867367] Key type ._llcrypt registered [12848.869552] Key type .llcrypt registered [12848.988934] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [12850.318731] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12850.358864] alg: No test for adler32 (adler32-zlib) [12851.363186] Lustre: Lustre: Build Version: 2.17.57_110_g5f5d390 [12851.600803] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [12853.247167] Key type lgssc registered [12854.300583] Lustre: Echo OBD driver; http://www.lustre.org/ [12879.487475] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12879.532383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12880.749535] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12880.806714] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12880.910810] Lustre: lustre-MDT0000: new disk, initializing [12880.997809] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12881.020174] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12884.543823] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [12893.894764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12894.016263] Lustre: 443889:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12894.067567] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12894.071980] Lustre: Skipped 1 previous similar message [12894.153294] Lustre: lustre-MDT0001: new disk, initializing [12894.279240] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12894.341777] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12894.388386] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12899.194826] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [12907.816229] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12913.370148] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12920.235211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12920.505827] Lustre: lustre-OST0000: new disk, initializing [12920.517884] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12920.527157] Lustre: 445858:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12920.610087] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12926.253874] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [12929.564113] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12929.580503] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12929.684189] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12934.836404] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12945.381875] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12945.385798] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12945.407954] Lustre: Skipped 2 previous similar messages [12945.428114] Lustre: Skipped 2 previous similar messages [12948.579919] Lustre: server umount lustre-MDT0000 complete [12952.016746] LustreError: 443880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789794060 with bad export cookie 18369980168052225482 [12952.019564] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12952.030279] LustreError: 443880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12952.520305] Lustre: server umount lustre-MDT0001 complete [12966.388716] Lustre: server umount lustre-OST0000 complete [12975.961992] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 01:01:23 (1789794083) [12983.153105] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12983.631681] LustreError: 447898:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12983.651186] LustreError: 447898:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12983.713443] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12987.512716] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [12988.897538] LustreError: 447899:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12992.996805] LustreError: 447898:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12995.219919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12995.504697] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12999.211665] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [13006.191871] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13010.864638] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13029.942070] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13030.222261] Lustre: Found index 0 for lustre-OST0000, updating log [13030.365485] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13035.841258] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [13042.956666] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13054.956054] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13065.191133] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13065.193596] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13065.204398] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13066.274621] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13066.282444] Lustre: Skipped 1 previous similar message [13069.547438] Lustre: server umount lustre-OST0000 complete [13084.503849] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13084.522181] Lustre: Skipped 1 previous similar message [13084.527125] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13084.532092] Lustre: Skipped 1 previous similar message [13088.160688] LNet: 1 local NIs in recovery (showing 1): 192.168.203.145@tcp [13089.761216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13089.771320] Lustre: Skipped 1 previous similar message [13090.795991] Lustre: server umount lustre-MDT0000 complete [13092.836215] LustreError: 447899:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13092.865820] LustreError: 447899:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13094.193710] LustreError: 447879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789794202 with bad export cookie 18369980168052226875 [13094.198113] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13094.201728] LustreError: 447879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13094.465677] Lustre: server umount lustre-MDT0001 complete [13102.449544] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 01:03:29 (1789794209) [13153.695397] LNet: 1 local NIs in recovery (showing 1): 192.168.203.145@tcp [13725.310918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13725.729594] LustreError: 454333:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13725.757147] LustreError: 454333:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13725.798501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13730.445188] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [13730.790851] LustreError: 454334:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13735.910134] LustreError: 454333:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13739.190897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13739.624732] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13744.092295] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [13752.296843] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13758.581114] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13778.995703] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13779.242113] Lustre: Found index 0 for lustre-OST0000, updating log [13779.516918] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13780.386640] LNet: 1 local NIs in recovery (showing 1): 192.168.203.145@tcp [13786.588309] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [13796.486235] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13798.842334] Lustre: server umount lustre-OST0000 complete [13803.875683] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13803.877409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13803.907655] Lustre: Skipped 2 previous similar messages [13803.916357] Lustre: Skipped 1 previous similar message [13804.964450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13804.972691] Lustre: Skipped 1 previous similar message [13807.021128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13807.034941] Lustre: Skipped 3 previous similar messages [13811.106353] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13811.115391] Lustre: Skipped 7 previous similar messages [13817.311123] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13817.428403] Lustre: server umount lustre-MDT0000 complete [13818.274264] LustreError: 455061:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13818.299056] LustreError: 455061:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13820.925256] LustreError: 454313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789794928 with bad export cookie 18369980168052227960 [13820.926811] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13820.937699] LustreError: 454313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13821.242841] Lustre: server umount lustre-MDT0001 complete [13840.799588] LNet: 1 local NIs in recovery (showing 1): 192.168.203.145@tcp [13851.375518] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [13860.665613] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [13894.724407] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 01:16:41 (1789795001) [13910.469783] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [13912.985639] Key type lgssc unregistered [13913.224583] LNet: 460944:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13913.232924] LNetError: 460944:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14042.278697] LNet: Removed LNI 192.168.203.145@tcp [14042.952603] Key type .llcrypt unregistered [14042.954294] Key type ._llcrypt unregistered [14055.414977] Key type ._llcrypt registered [14055.417142] Key type .llcrypt registered [14055.525784] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [14056.153839] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14056.167426] alg: No test for adler32 (adler32-zlib) [14057.115398] Lustre: Lustre: Build Version: 2.17.57_110_g5f5d390 [14057.320783] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [14058.986084] Key type lgssc registered [14059.867621] Lustre: Echo OBD driver; http://www.lustre.org/ [14062.826306] LNet: Added LNI 192.168.203.145@tcp100 [8/256/0/180] [14063.525296] LNet: Added LNI 192.168.203.145@tcp101 [8/256/0/180] [14064.116553] LNet: Added LNI 192.168.203.145@tcp102 [8/256/0/180] [14064.593701] LNet: Added LNI 192.168.203.145@tcp103 [8/256/0/180] [14065.253364] LNet: Added LNI 192.168.203.145@tcp104 [8/256/0/180] [14065.803189] LNet: Added LNI 192.168.203.145@tcp105 [8/256/0/180] [14066.387273] LNet: Added LNI 192.168.203.145@tcp106 [8/256/0/180] [14066.983648] LNet: Added LNI 192.168.203.145@tcp107 [8/256/0/180] [14067.575863] LNet: Added LNI 192.168.203.145@tcp108 [8/256/0/180] [14068.159329] LNet: Added LNI 192.168.203.145@tcp109 [8/256/0/180] [14068.780296] LNet: Added LNI 192.168.203.145@tcp110 [8/256/0/180] [14069.526226] LNet: Added LNI 192.168.203.145@tcp111 [8/256/0/180] [14070.141883] LNet: Added LNI 192.168.203.145@tcp112 [8/256/0/180] [14070.811062] LNet: Added LNI 192.168.203.145@tcp113 [8/256/0/180] [14071.384584] LNet: Added LNI 192.168.203.145@tcp114 [8/256/0/180] [14072.012944] LNet: Added LNI 192.168.203.145@tcp115 [8/256/0/180] [14072.697506] LNet: Added LNI 192.168.203.145@tcp116 [8/256/0/180] [14073.515111] LNet: Added LNI 192.168.203.145@tcp117 [8/256/0/180] [14074.298458] LNet: Added LNI 192.168.203.145@tcp118 [8/256/0/180] [14075.040707] LNet: Added LNI 192.168.203.145@tcp119 [8/256/0/180] [14075.749428] LNet: Added LNI 192.168.203.145@tcp120 [8/256/0/180] [14076.467147] LNet: Added LNI 192.168.203.145@tcp121 [8/256/0/180] [14077.175790] LNet: Added LNI 192.168.203.145@tcp122 [8/256/0/180] [14077.858409] LNet: Added LNI 192.168.203.145@tcp123 [8/256/0/180] [14078.539558] LNet: Added LNI 192.168.203.145@tcp124 [8/256/0/180] [14079.181048] LNet: Added LNI 192.168.203.145@tcp125 [8/256/0/180] [14079.777184] LNet: Added LNI 192.168.203.145@tcp126 [8/256/0/180] [14080.357703] LNet: Added LNI 192.168.203.145@tcp127 [8/256/0/180] [14080.984548] LNet: Added LNI 192.168.203.145@tcp128 [8/256/0/180] [14081.632536] LNet: Added LNI 192.168.203.145@tcp129 [8/256/0/180] [14082.299897] LNet: Added LNI 192.168.203.145@tcp130 [8/256/0/180] [14082.910634] LNet: Added LNI 192.168.203.145@tcp131 [8/256/0/180] [14083.511504] LNet: Added LNI 192.168.203.145@tcp132 [8/256/0/180] [14084.172655] LNet: Added LNI 192.168.203.145@tcp133 [8/256/0/180] [14084.890853] LNet: Added LNI 192.168.203.145@tcp134 [8/256/0/180] [14085.603119] LNet: Added LNI 192.168.203.145@tcp135 [8/256/0/180] [14090.797088] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14090.854438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14091.951897] Lustre: 463951:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.145@tcp: can't add NID 192.168.203.145@tcp130: rc = -75 [14092.011622] LNetError: 463977:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.145 [14092.022630] LNetError: Protocol error connecting to 192.168.203.145@tcp118 at host 192.168.203.145:988: is it running a compatible version of Lustre? [14092.038531] LNetError: Skipped 7 previous similar messages [14092.102159] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14092.127514] LNet: 6 local NIs in recovery (showing 6): 192.168.203.145@tcp125, 192.168.203.145@tcp127, 192.168.203.145@tcp129, 192.168.203.145@tcp126, 192.168.203.145@tcp118, 192.168.203.145@tcp128 [14092.131732] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14092.218953] Lustre: lustre-MDT0000: new disk, initializing [14092.299855] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14092.316234] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14093.088726] LNetError: 463971:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.145 [14093.097752] LNetError: 463971:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 12 previous similar messages [14093.101562] LNetError: Protocol error connecting to 192.168.203.145@tcp114 at host 192.168.203.145:988: is it running a compatible version of Lustre? [14093.116110] LNetError: Skipped 2 previous similar messages [14095.202912] LNetError: 463971:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.145 [14095.220788] LNetError: 463971:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 6 previous similar messages [14095.229481] LNetError: Protocol error connecting to 192.168.203.145@tcp125 at host 192.168.203.145:988: is it running a compatible version of Lustre? [14095.235948] LNetError: Skipped 6 previous similar messages [14095.403277] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [14099.299399] LNetError: 463983:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.145 [14099.317774] LNetError: Protocol error connecting to 192.168.203.145@tcp125 at host 192.168.203.145:988: is it running a compatible version of Lustre? [14104.296847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14104.437498] Lustre: 463968:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14104.515712] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14104.527467] Lustre: Skipped 1 previous similar message [14104.605505] Lustre: lustre-MDT0001: new disk, initializing [14104.648324] Lustre: 464771:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.145@tcp: can't add NID 192.168.203.145@tcp130: rc = -75 [14104.668763] Lustre: 464771:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14104.773513] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14104.804819] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14104.813639] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14106.770109] Lustre: 464934:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.145@tcp: can't add NID 192.168.203.145@tcp130: rc = -75 [14106.776474] Lustre: 464934:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14107.488392] LNetError: 463976:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.145 [14107.494218] LNetError: Protocol error connecting to 192.168.203.145@tcp125 at host 192.168.203.145:988: is it running a compatible version of Lustre? [14108.075219] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [14115.299703] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14119.734928] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14121.826900] LNetError: 461410:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.145 [14121.843710] LNetError: Protocol error connecting to 192.168.203.145@tcp109 at host 192.168.203.145:988: is it running a compatible version of Lustre? [14125.065905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14125.373413] Lustre: lustre-OST0000: new disk, initializing [14125.377538] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14125.386355] Lustre: 465944:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14125.483262] Lustre: 465949:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.145@tcp: can't add NID 192.168.203.145@tcp130: rc = -75 [14125.490666] Lustre: 465949:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14125.530530] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14130.592284] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [14135.355841] Lustre: 466516:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.145@tcp: can't add NID 192.168.203.145@tcp130: rc = -75 [14135.376821] Lustre: 466516:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14135.439079] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14135.448933] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14135.591376] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14138.212570] LNetError: 461410:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.145 [14138.226196] LNetError: 461410:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 11 previous similar messages [14138.233512] LNetError: Protocol error connecting to 192.168.203.145@tcp123 at host 192.168.203.145:988: is it running a compatible version of Lustre? [14138.246751] LNetError: Skipped 11 previous similar messages [14138.473916] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14146.024315] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14146.025071] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14146.035115] LustreError: Skipped 1 previous similar message [14146.036652] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14146.047552] Lustre: Skipped 1 previous similar message [14149.710780] Lustre: server umount lustre-OST0000 complete [14152.543247] LNet: 1 local NIs in recovery (showing 1): 192.168.203.145@tcp125 [14156.256875] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14156.259651] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14156.264798] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14156.280191] Lustre: Skipped 1 previous similar message [14156.290276] Lustre: Skipped 2 previous similar messages [14158.786460] Lustre: server umount lustre-MDT0000 complete [14161.920977] LustreError: 463960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789795269 with bad export cookie 8838819039808393252 [14161.927806] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14161.942967] LustreError: 463960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14162.197817] Lustre: server umount lustre-MDT0001 complete [14170.314726] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [14172.239233] Key type lgssc unregistered [14172.504771] LNet: 467853:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14172.510278] LNetError: 467853:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14172.514609] LNet: Removed LNI 192.168.203.145@tcp [14173.535659] LNet: Removed LNI 192.168.203.145@tcp100 [14173.540527] LNetError: 467853:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14173.552984] LNetError: 467853:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14173.558379] LNet: Removed LNI 192.168.203.145@tcp101 [14174.623192] LNet: Removed LNI 192.168.203.145@tcp102 [14174.625415] LNetError: 467853:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14174.634958] LNetError: 467853:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14174.644198] LNet: Removed LNI 192.168.203.145@tcp103 [14175.713334] LNet: Removed LNI 192.168.203.145@tcp104 [14176.736230] LNet: Removed LNI 192.168.203.145@tcp105 [14176.741960] LNetError: 467853:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14176.749165] LNetError: 467853:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14176.760102] LNet: Removed LNI 192.168.203.145@tcp106 [14177.823332] LNet: Removed LNI 192.168.203.145@tcp107 [14178.847160] LNet: Removed LNI 192.168.203.145@tcp108 [14179.871347] LNet: Removed LNI 192.168.203.145@tcp109 [14180.896149] LNet: Removed LNI 192.168.203.145@tcp110 [14180.899773] LNetError: 467853:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14180.903443] LNetError: 467853:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14180.913498] LNet: Removed LNI 192.168.203.145@tcp111 [14181.919304] LNet: Removed LNI 192.168.203.145@tcp112 [14182.943222] LNet: Removed LNI 192.168.203.145@tcp113 [14183.967294] LNet: Removed LNI 192.168.203.145@tcp114 [14184.991357] LNet: Removed LNI 192.168.203.145@tcp115 [14186.015183] LNet: Removed LNI 192.168.203.145@tcp116 [14187.039703] LNet: Removed LNI 192.168.203.145@tcp117 [14188.063164] LNet: Removed LNI 192.168.203.145@tcp118 [14189.087406] LNet: Removed LNI 192.168.203.145@tcp119 [14189.092158] LNetError: 467853:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14189.106158] LNetError: 467853:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [14189.117661] LNet: Removed LNI 192.168.203.145@tcp120 [14190.176647] LNet: Removed LNI 192.168.203.145@tcp121 [14191.199187] LNet: Removed LNI 192.168.203.145@tcp122 [14192.223163] LNet: Removed LNI 192.168.203.145@tcp123 [14193.247339] LNet: Removed LNI 192.168.203.145@tcp124 [14193.252210] LNet: Removed LNI 192.168.203.145@tcp125 [14194.273276] LNet: Removed LNI 192.168.203.145@tcp126 [14195.295134] LNet: Removed LNI 192.168.203.145@tcp127 [14196.319458] LNet: Removed LNI 192.168.203.145@tcp128 [14197.343221] LNet: Removed LNI 192.168.203.145@tcp129 [14197.345434] LNet: Removed LNI 192.168.203.145@tcp130 [14197.350079] LNet: Removed LNI 192.168.203.145@tcp131 [14197.353528] LNet: Removed LNI 192.168.203.145@tcp132 [14197.361547] LNet: Removed LNI 192.168.203.145@tcp133 [14197.364189] LNet: Removed LNI 192.168.203.145@tcp134 [14197.384358] LNet: Removed LNI 192.168.203.145@tcp135 [14197.959232] Key type .llcrypt unregistered [14197.961528] Key type ._llcrypt unregistered [14214.519884] Key type ._llcrypt registered [14214.521894] Key type .llcrypt registered [14214.584670] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [14225.836052] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [14226.488400] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14226.533702] alg: No test for adler32 (adler32-zlib) [14227.522987] Lustre: Lustre: Build Version: 2.17.57_110_g5f5d390 [14227.707680] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [14229.343354] Key type lgssc registered [14230.351948] Lustre: Echo OBD driver; http://www.lustre.org/ [14259.005898] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 01:22:46 (1789795366) [14272.926527] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [14274.870710] Key type lgssc unregistered [14275.071877] LNet: 471913:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14275.077284] LNetError: 471913:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14275.095284] LNet: Removed LNI 192.168.203.145@tcp [14275.639625] Key type .llcrypt unregistered [14275.641683] Key type ._llcrypt unregistered [14287.887467] Key type ._llcrypt registered [14287.890742] Key type .llcrypt registered [14287.976285] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [14288.890102] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14288.905852] alg: No test for adler32 (adler32-zlib) [14289.885525] Lustre: Lustre: Build Version: 2.17.57_110_g5f5d390 [14290.048278] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [14291.751183] Key type lgssc registered [14292.645252] Lustre: Echo OBD driver; http://www.lustre.org/ [14299.667077] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14299.705548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14300.940080] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14300.963374] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14301.027450] Lustre: lustre-MDT0000: new disk, initializing [14301.114738] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14301.129624] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14304.462917] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [14312.846937] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14312.896724] Lustre: 473101:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14312.942148] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14312.946237] Lustre: Skipped 1 previous similar message [14313.018524] Lustre: lustre-MDT0001: new disk, initializing [14313.091644] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14313.112613] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14313.117105] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14316.742653] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [14324.148918] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14328.730692] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14333.704117] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14333.882404] Lustre: lustre-OST0000: new disk, initializing [14333.886684] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14333.896930] Lustre: 475066:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14333.975692] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14338.775203] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [14341.158700] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14341.167663] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14341.310796] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14346.198781] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14348.839897] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing lnet_if_list [14352.256370] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14352.280530] LNet: Added LNI 192.168.203.145@tcp10 [8/256/0/180] [14355.899972] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14355.937542] LNet: Added LNI 192.168.203.145@tcp11 [8/256/0/180] [14359.384943] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14359.410697] LNet: Added LNI 192.168.203.145@tcp12 [8/256/0/180] [14362.406163] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14362.429414] LNet: Added LNI 192.168.203.145@tcp13 [8/256/0/180] [14365.521417] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14365.547621] LNet: Added LNI 192.168.203.145@tcp14 [8/256/0/180] [14368.364209] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14368.386599] LNet: Added LNI 192.168.203.145@tcp15 [8/256/0/180] [14371.677717] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14371.702293] LNet: Added LNI 192.168.203.145@tcp16 [8/256/0/180] [14374.687162] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14374.715612] LNet: Added LNI 192.168.203.145@tcp17 [8/256/0/180] [14378.251348] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14378.276421] LNet: Added LNI 192.168.203.145@tcp18 [8/256/0/180] [14381.953703] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14381.973452] LNet: Added LNI 192.168.203.145@tcp19 [8/256/0/180] [14385.716495] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14385.736145] LNet: Added LNI 192.168.203.145@tcp20 [8/256/0/180] [14389.363581] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14389.403429] LNet: Added LNI 192.168.203.145@tcp21 [8/256/0/180] [14393.528981] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14393.558613] LNet: Added LNI 192.168.203.145@tcp22 [8/256/0/180] [14397.485675] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14397.522399] LNet: Added LNI 192.168.203.145@tcp23 [8/256/0/180] [14401.927510] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14401.969140] LNet: Added LNI 192.168.203.145@tcp24 [8/256/0/180] [14406.403215] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14406.425389] LNet: Added LNI 192.168.203.145@tcp25 [8/256/0/180] [14410.780135] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14410.799776] LNet: Added LNI 192.168.203.145@tcp26 [8/256/0/180] [14415.782472] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14415.814639] LNet: Added LNI 192.168.203.145@tcp27 [8/256/0/180] [14420.547593] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14420.569798] LNet: Added LNI 192.168.203.145@tcp28 [8/256/0/180] [14424.812895] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14424.833174] LNet: Added LNI 192.168.203.145@tcp29 [8/256/0/180] [14429.356408] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14429.378586] LNet: Added LNI 192.168.203.145@tcp30 [8/256/0/180] [14433.435330] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14433.451585] LNet: Added LNI 192.168.203.145@tcp31 [8/256/0/180] [14437.536954] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14437.560529] LNet: Added LNI 192.168.203.145@tcp32 [8/256/0/180] [14441.907907] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14441.945231] LNet: Added LNI 192.168.203.145@tcp33 [8/256/0/180] [14447.010479] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14447.037943] LNet: Added LNI 192.168.203.145@tcp34 [8/256/0/180] [14451.759955] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14451.776278] LNet: Added LNI 192.168.203.145@tcp35 [8/256/0/180] [14457.012747] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14457.047107] LNet: Added LNI 192.168.203.145@tcp36 [8/256/0/180] [14462.793224] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14462.818578] LNet: Added LNI 192.168.203.145@tcp37 [8/256/0/180] [14468.159881] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14468.183214] LNet: Added LNI 192.168.203.145@tcp38 [8/256/0/180] [14473.583820] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14473.604739] LNet: Added LNI 192.168.203.145@tcp39 [8/256/0/180] [14478.806325] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14478.827078] LNet: Added LNI 192.168.203.145@tcp40 [8/256/0/180] [14484.170963] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14484.200414] LNet: Added LNI 192.168.203.145@tcp41 [8/256/0/180] [14489.395814] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [14489.415196] LNet: Added LNI 192.168.203.145@tcp42 [8/256/0/180] [14495.237810] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [14495.267641] LNet: Added LNI 192.168.203.145@tcp43 [8/256/0/180] [14500.314876] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [14500.342478] LNet: Added LNI 192.168.203.145@tcp44 [8/256/0/180] [14505.815759] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [14505.848613] LNet: Added LNI 192.168.203.145@tcp45 [8/256/0/180] [14511.820897] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [14511.856515] LNet: Added LNI 192.168.203.145@tcp46 [8/256/0/180] [14517.732367] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [14517.762920] LNet: Added LNI 192.168.203.145@tcp47 [8/256/0/180] [14523.779800] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [14523.802206] LNet: Added LNI 192.168.203.145@tcp48 [8/256/0/180] [14529.950186] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [14529.977581] LNet: Added LNI 192.168.203.145@tcp49 [8/256/0/180] [14536.053110] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [14536.075835] LNet: Added LNI 192.168.203.145@tcp50 [8/256/0/180] [14551.382194] Lustre: server umount lustre-MDT0000 complete [14551.523849] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14551.536309] Lustre: Skipped 1 previous similar message [14553.849304] LustreError: 473093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789795661 with bad export cookie 17122726997797446060 [14553.853454] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14553.857351] LustreError: 473093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14554.040212] Lustre: server umount lustre-MDT0001 complete [14567.040829] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [14572.003085] Lustre: 472372:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789795664/real 1789795664] req@00000000fae8c372 x1876736499387520/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789795680 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14572.022124] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14572.036361] Lustre: Skipped 1 previous similar message [14573.427082] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14574.113167] Lustre: 472372:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789795666/real 1789795666] req@00000000df0b6e2a x1876736499387648/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789795682 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14577.119265] Lustre: 472374:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789795669/real 1789795669] req@00000000423ea0fb x1876736499387776/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789795685 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14578.207562] LustreError: 472371:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000090b4f0cc x1876736499388672/t0(0) o250->MGC192.168.203.145@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14578.435909] LustreError: 475060:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14578.449765] LustreError: 475060:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [14578.503629] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14579.487760] Lustre: 472375:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789795671/real 1789795671] req@00000000673b3e85 x1876736499388032/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789795687 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14582.313023] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [14582.626145] LustreError: 499272:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14582.643177] LustreError: 499272:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14584.671165] Lustre: 472373:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789795676/real 1789795676] req@00000000455a8cbe x1876736499388160/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789795692 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14587.872458] LustreError: 475060:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14587.885026] LustreError: 475060:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14588.346623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14588.592959] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14591.435712] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [14591.653203] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14596.491859] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14600.047959] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14614.496629] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14614.497590] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14614.505829] LustreError: Skipped 1 previous similar message [14614.508751] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14614.523220] Lustre: Skipped 1 previous similar message [14615.726435] Lustre: server umount lustre-OST0000 complete [14619.619503] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14619.625982] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14619.644378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14619.650812] Lustre: Skipped 1 previous similar message [14622.689854] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14622.695132] Lustre: Skipped 1 previous similar message [14624.384975] Lustre: server umount lustre-MDT0000 complete [14626.966364] LustreError: 473091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789795735 with bad export cookie 17122726997797448517 [14626.967677] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14626.972620] LustreError: 473091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14627.223562] Lustre: server umount lustre-MDT0001 complete [14635.494211] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [14637.599759] Key type lgssc unregistered [14637.882398] LNet: 502047:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14637.889926] LNetError: 502047:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14637.894892] LNet: Removed LNI 192.168.203.145@tcp [14637.897228] LNet: Removed LNI 192.168.203.145@tcp10 [14637.901250] LNet: Removed LNI 192.168.203.145@tcp11 [14637.904728] LNet: Removed LNI 192.168.203.145@tcp12 [14637.907264] LNet: Removed LNI 192.168.203.145@tcp13 [14637.910077] LNet: Removed LNI 192.168.203.145@tcp14 [14637.912665] LNet: Removed LNI 192.168.203.145@tcp15 [14637.915432] LNet: Removed LNI 192.168.203.145@tcp16 [14637.919498] LNet: Removed LNI 192.168.203.145@tcp17 [14637.921778] LNet: Removed LNI 192.168.203.145@tcp18 [14637.926318] LNet: Removed LNI 192.168.203.145@tcp19 [14637.928515] LNet: Removed LNI 192.168.203.145@tcp20 [14637.931312] LNet: Removed LNI 192.168.203.145@tcp21 [14637.934275] LNet: Removed LNI 192.168.203.145@tcp22 [14637.935907] LNet: Removed LNI 192.168.203.145@tcp23 [14637.938115] LNet: Removed LNI 192.168.203.145@tcp24 [14637.940342] LNet: Removed LNI 192.168.203.145@tcp25 [14637.942488] LNet: Removed LNI 192.168.203.145@tcp26 [14637.944666] LNet: Removed LNI 192.168.203.145@tcp27 [14637.946880] LNet: Removed LNI 192.168.203.145@tcp28 [14637.949706] LNet: Removed LNI 192.168.203.145@tcp29 [14637.952331] LNet: Removed LNI 192.168.203.145@tcp30 [14637.954879] LNet: Removed LNI 192.168.203.145@tcp31 [14637.956771] LNet: Removed LNI 192.168.203.145@tcp32 [14637.958676] LNet: Removed LNI 192.168.203.145@tcp33 [14637.961259] LNet: Removed LNI 192.168.203.145@tcp34 [14637.963558] LNet: Removed LNI 192.168.203.145@tcp35 [14637.965335] LNet: Removed LNI 192.168.203.145@tcp36 [14637.967198] LNet: Removed LNI 192.168.203.145@tcp37 [14637.969451] LNet: Removed LNI 192.168.203.145@tcp38 [14637.971732] LNet: Removed LNI 192.168.203.145@tcp39 [14637.974203] LNet: Removed LNI 192.168.203.145@tcp40 [14637.976615] LNet: Removed LNI 192.168.203.145@tcp41 [14637.979219] LNet: Removed LNI 192.168.203.145@tcp42 [14637.981525] LNet: Removed LNI 192.168.203.145@tcp43 [14637.983394] LNet: Removed LNI 192.168.203.145@tcp44 [14637.985680] LNet: Removed LNI 192.168.203.145@tcp45 [14637.987769] LNet: Removed LNI 192.168.203.145@tcp46 [14637.989914] LNet: Removed LNI 192.168.203.145@tcp47 [14637.992161] LNet: Removed LNI 192.168.203.145@tcp48 [14637.994214] LNet: Removed LNI 192.168.203.145@tcp49 [14638.008487] LNet: Removed LNI 192.168.203.145@tcp50 [14638.528324] Key type .llcrypt unregistered [14638.531119] Key type ._llcrypt unregistered [14654.106707] Key type ._llcrypt registered [14654.110162] Key type .llcrypt registered [14654.214284] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [14665.174112] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [14665.765746] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14665.781187] alg: No test for adler32 (adler32-zlib) [14666.753230] Lustre: Lustre: Build Version: 2.17.57_110_g5f5d390 [14666.960873] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [14668.663215] Key type lgssc registered [14669.481149] Lustre: Echo OBD driver; http://www.lustre.org/ [14698.473375] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14698.495219] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14699.694951] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14699.720280] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14699.783424] Lustre: lustre-MDT0000: new disk, initializing [14699.880182] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14699.892481] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14703.708974] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [14712.535555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14712.636262] Lustre: 505219:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14712.704857] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14712.708066] Lustre: Skipped 1 previous similar message [14712.837619] Lustre: lustre-MDT0001: new disk, initializing [14712.958194] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14712.998129] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14713.010701] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14716.361925] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [14723.504549] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14728.072480] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14733.732892] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14733.878708] Lustre: *** cfs_fail_loc=90f, val=0*** [14733.958690] Lustre: lustre-OST0000: new disk, initializing [14733.964072] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14733.972163] Lustre: 507232:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14734.040622] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14739.154186] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [14743.089217] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14743.096214] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14743.175408] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14747.038938] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14753.763696] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14753.766124] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14753.770024] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14753.772393] Lustre: Skipped 1 previous similar message [14753.791966] Lustre: Skipped 1 previous similar message [14758.543946] Lustre: server umount lustre-OST0000 complete [14764.000619] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14764.006358] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14764.015617] LustreError: Skipped 1 previous similar message [14764.047956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14764.056304] Lustre: Skipped 1 previous similar message [14767.965215] Lustre: server umount lustre-MDT0000 complete [14770.808778] LustreError: 505210:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789795878 with bad export cookie 2581687295062411098 [14770.820735] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14770.822777] LustreError: 505210:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14771.059903] Lustre: server umount lustre-MDT0001 complete [14779.365671] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [14781.768269] Key type lgssc unregistered [14782.007937] LNet: 509136:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14782.015815] LNetError: 509136:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14782.039032] LNet: Removed LNI 192.168.203.145@tcp [14782.724443] Key type .llcrypt unregistered [14782.726265] Key type ._llcrypt unregistered [14797.535497] Key type ._llcrypt registered [14797.537538] Key type .llcrypt registered [14797.629180] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [14798.411559] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14798.517428] alg: No test for adler32 (adler32-zlib) [14799.502598] Lustre: Lustre: Build Version: 2.17.57_110_g5f5d390 [14799.699213] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [14801.327172] Key type lgssc registered [14802.314712] Lustre: Echo OBD driver; http://www.lustre.org/ [14808.407448] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14808.423492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14809.760293] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14813.080956] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [14819.301826] LustreError: 510384:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14819.321543] LustreError: 510384:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [14819.350150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14819.648507] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14822.558710] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [14827.767161] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14832.032271] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14836.709095] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14836.992757] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14841.458859] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [14846.779581] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14864.845741] loop0: detected capacity change from 0 to 204800000 [14864.946847] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14864.953477] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14864.961490] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14868.744902] loop0: detected capacity change from 0 to 204800000 [14868.944962] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14868.957245] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14868.973446] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14872.438137] loop0: detected capacity change from 0 to 204800000 [14876.118287] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14876.197297] Lustre: fs15246-OST0000: new disk, initializing [14876.201658] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [14876.205709] Lustre: 513997:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14876.238207] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14880.133144] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [14887.395541] loop1: detected capacity change from 0 to 204800000 [14891.276696] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14891.367430] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [14891.428701] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [14891.486589] Lustre: fs15246-MDT0000: new disk, initializing [14891.628441] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14891.675679] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14891.791956] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [14892.834070] Lustre: 515118:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000df2a5ca5 x1876737033019904/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1789796016 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [14895.028575] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [14901.921928] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [14901.928317] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [14901.949311] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [14910.440235] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14910.454983] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [14915.557667] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [14916.162142] Lustre: server umount fs15246-MDT0000 complete [14920.676775] LustreError: 510378:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14921.665887] Lustre: server umount fs15246-OST0000 complete [14930.916733] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14930.920661] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14930.927081] Lustre: Skipped 1 previous similar message [14930.937390] Lustre: Skipped 1 previous similar message [14935.206061] Lustre: server umount lustre-OST0000 complete [14941.155636] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14941.169813] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14941.177827] Lustre: Skipped 1 previous similar message [14941.187265] Lustre: Skipped 1 previous similar message [14944.148811] Lustre: server umount lustre-MDT0000 complete [14946.285177] LustreError: 510377:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14946.297656] LustreError: 510377:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14946.946367] LustreError: 510363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789796054 with bad export cookie 1091947086349267009 [14946.957803] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14946.960644] LustreError: 510363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14947.255212] Lustre: server umount lustre-MDT0001 complete [14955.748649] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [14958.024173] Key type lgssc unregistered [14958.244580] LNet: 518215:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14958.250186] LNetError: 518215:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14958.258670] LNet: Removed LNI 192.168.203.145@tcp [14958.902662] Key type .llcrypt unregistered [14958.907575] Key type ._llcrypt unregistered [14971.402686] Key type ._llcrypt registered [14971.405064] Key type .llcrypt registered [14971.465663] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [14972.055260] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14972.106223] alg: No test for adler32 (adler32-zlib) [14973.037746] Lustre: Lustre: Build Version: 2.17.57_110_g5f5d390 [14973.199767] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [14974.815157] Key type lgssc registered [14975.549899] Lustre: Echo OBD driver; http://www.lustre.org/ [15002.702827] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [15009.522548] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [15033.082841] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15033.102797] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15034.321236] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15034.348786] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15034.411912] Lustre: lustre-MDT0000: new disk, initializing [15034.478677] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15034.501124] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15037.066329] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [15044.214741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15044.284386] Lustre: 522653:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15044.320373] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15044.324793] Lustre: Skipped 1 previous similar message [15044.381419] Lustre: lustre-MDT0001: new disk, initializing [15044.441276] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15044.463711] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15044.468556] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15047.346317] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [15054.108870] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15057.750348] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15062.423415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15062.633381] Lustre: lustre-OST0000: new disk, initializing [15062.639413] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15062.643331] Lustre: 524617:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15062.701512] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15066.609410] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [15068.689976] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15068.698472] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15068.765684] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15072.891307] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15075.635938] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15075.766361] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15078.081486] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15078.227424] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15084.002620] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15084.006813] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15084.010251] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15084.014978] Lustre: Skipped 1 previous similar message [15085.433504] Lustre: server umount lustre-OST0000 complete [15089.123716] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15089.130685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15089.134230] Lustre: Skipped 1 previous similar message [15089.146088] Lustre: Skipped 2 previous similar messages [15094.060414] Lustre: server umount lustre-MDT0000 complete [15096.376201] LustreError: 522643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789796204 with bad export cookie 12360130220643820192 [15096.379696] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15096.386407] LustreError: 522643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15096.611868] Lustre: server umount lustre-MDT0001 complete [15111.875902] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [15118.321950] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [15136.226947] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14848 sec ========= 01:37:24 (1789796244) [15137.230121] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 01:37:25 (1789796245) === [15139.362458] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 01:37:27 (1789796247) === [15161.933723] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [15163.558569] Key type lgssc unregistered [15163.773434] LNet: 531250:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15163.778841] LNetError: 531250:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15163.795732] LNet: Removed LNI 192.168.203.145@tcp [15164.351694] Key type .llcrypt unregistered [15164.353891] Key type ._llcrypt unregistered