[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 525616854 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003147] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.008754] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010014] pid_max: default: 32768 minimum: 301 [ 0.012121] LSM: Security Framework initializing [ 0.013063] Yama: becoming mindful. [ 0.014042] SELinux: Initializing. [ 0.015071] *** VALIDATE selinux *** [ 0.023180] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028152] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029149] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030099] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031108] *** VALIDATE tmpfs *** [ 0.033369] *** VALIDATE proc *** [ 0.034260] *** VALIDATE cgroup *** [ 0.035010] *** VALIDATE cgroup2 *** [ 0.036255] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037156] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039029] Spectre V2 : User space: Vulnerable [ 0.040006] Speculative Store Bypass: Vulnerable [ 0.042664] debug: unmapping init [mem 0xffffffffb9659000-0xffffffffb9660fff] [ 0.044186] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045705] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046025] ... version: 2 [ 0.047012] ... bit width: 48 [ 0.048013] ... generic registers: 4 [ 0.049015] ... value mask: 0000ffffffffffff [ 0.050011] ... max period: 00007fffffffffff [ 0.051015] ... fixed-purpose events: 3 [ 0.052013] ... event mask: 000000070000000f [ 0.053251] rcu: Hierarchical SRCU implementation. [ 0.055404] smp: Bringing up secondary CPUs ... [ 0.056609] x86: Booting SMP configuration: [ 0.057022] .... node #0, CPUs: #1 #2 #3 [ 0.060175] smp: Brought up 1 node, 4 CPUs [ 0.062014] smpboot: Max logical packages: 1 [ 0.063015] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.136874] node 0 deferred pages initialised in 72ms [ 0.141088] devtmpfs: initialized [ 0.142259] x86/mm: Memory block size: 128MB [ 0.144774] gcov: version magic: 0x41383552 [ 0.148016] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.149072] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.150265] pinctrl core: initialized pinctrl subsystem [ 0.152254] [ 0.152808] ************************************************************* [ 0.155013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.158012] ** ** [ 0.159012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.162009] ** ** [ 0.164008] ** This means that this kernel is built to expose internal ** [ 0.165010] ** IOMMU data structures, which may compromise security on ** [ 0.168014] ** your system. ** [ 0.171023] ** ** [ 0.174020] ** If you see this message and you are not debugging the ** [ 0.177025] ** kernel, report this immediately to your vendor! ** [ 0.180035] ** ** [ 0.182021] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.185019] ************************************************************* [ 0.188700] NET: Registered protocol family 16 [ 0.190482] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.193066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.196125] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.199533] cpuidle: using governor menu [ 0.200401] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.202425] PCI: Using configuration type 1 for base access [ 0.205144] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.215095] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.216030] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.218082] cryptd: max_cpu_qlen set to 1000 [ 0.220296] ACPI: Added _OSI(Module Device) [ 0.222025] ACPI: Added _OSI(Processor Device) [ 0.224019] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.226030] ACPI: Added _OSI(Processor Aggregator Device) [ 0.232292] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.241616] ACPI: Interpreter enabled [ 0.242077] ACPI: PM: (supports S0 S3 S4 S5) [ 0.243011] ACPI: Using IOAPIC for interrupt routing [ 0.244133] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.245400] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.254924] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.257073] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.261032] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.265138] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.270554] acpiphp: Slot [2] registered [ 0.273215] acpiphp: Slot [5] registered [ 0.274200] acpiphp: Slot [6] registered [ 0.276181] acpiphp: Slot [7] registered [ 0.278172] acpiphp: Slot [8] registered [ 0.280189] acpiphp: Slot [9] registered [ 0.281189] acpiphp: Slot [10] registered [ 0.283213] acpiphp: Slot [3] registered [ 0.285136] acpiphp: Slot [4] registered [ 0.286109] acpiphp: Slot [11] registered [ 0.288097] acpiphp: Slot [12] registered [ 0.289105] acpiphp: Slot [13] registered [ 0.291158] acpiphp: Slot [14] registered [ 0.292129] acpiphp: Slot [15] registered [ 0.293131] acpiphp: Slot [16] registered [ 0.295119] acpiphp: Slot [17] registered [ 0.297208] acpiphp: Slot [18] registered [ 0.298122] acpiphp: Slot [19] registered [ 0.300115] acpiphp: Slot [20] registered [ 0.302109] acpiphp: Slot [21] registered [ 0.303133] acpiphp: Slot [22] registered [ 0.305129] acpiphp: Slot [23] registered [ 0.306094] acpiphp: Slot [24] registered [ 0.308116] acpiphp: Slot [25] registered [ 0.309084] acpiphp: Slot [26] registered [ 0.311119] acpiphp: Slot [27] registered [ 0.313118] acpiphp: Slot [28] registered [ 0.314142] acpiphp: Slot [29] registered [ 0.316135] acpiphp: Slot [30] registered [ 0.318168] acpiphp: Slot [31] registered [ 0.320092] PCI host bridge to bus 0000:00 [ 0.322031] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.325036] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.328037] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.330033] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.334036] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.337034] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.340225] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.343227] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.347495] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.357813] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.363852] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.368029] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.370025] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.372023] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.375438] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.378758] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.381050] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.383819] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.389022] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.400021] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.406019] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.412318] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.418019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.425023] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.443021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.455258] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.462018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.469020] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.489024] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.499515] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.507023] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.514022] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.534022] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.544166] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.552020] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.559017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.574022] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.584468] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.590020] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.597020] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.615026] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.627631] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.634017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.641016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.661020] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.672831] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.675542] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.678443] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.681385] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.684279] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.689054] iommu: Default domain type: Passthrough [ 0.691425] SCSI subsystem initialized [ 0.694193] ACPI: bus type USB registered [ 0.696110] usbcore: registered new interface driver usbfs [ 0.699158] usbcore: registered new interface driver hub [ 0.701115] usbcore: registered new device driver usb [ 0.704167] pps_core: LinuxPPS API ver. 1 registered [ 0.706022] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.709074] PTP clock support registered [ 0.711176] EDAC MC: Ver: 3.0.0 [ 0.713445] PCI: Using ACPI for IRQ routing [ 0.715899] NetLabel: Initializing [ 0.717013] NetLabel: domain hash size = 128 [ 0.718011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.720094] NetLabel: unlabeled traffic allowed by default [ 0.722088] vgaarb: loaded [ 0.723350] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.726020] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.732356] clocksource: Switched to clocksource kvm-clock [ 0.839656] VFS: Disk quotas dquot_6.6.0 [ 0.841466] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.844453] *** VALIDATE ramfs *** [ 0.845906] *** VALIDATE hugetlbfs *** [ 0.847944] pnp: PnP ACPI init [ 0.851405] pnp: PnP ACPI: found 6 devices [ 0.880225] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.883451] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.885585] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.887795] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.890325] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.893124] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.896320] NET: Registered protocol family 2 [ 0.898900] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.904938] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.908503] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.913389] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.916732] TCP: Hash tables configured (established 65536 bind 65536) [ 0.919716] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.922726] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.925713] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.929114] NET: Registered protocol family 1 [ 0.933238] RPC: Registered named UNIX socket transport module. [ 0.935485] RPC: Registered udp transport module. [ 0.937511] RPC: Registered tcp transport module. [ 0.939473] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.942375] NET: Registered protocol family 44 [ 0.946152] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.949104] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.951658] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.954675] PCI: CLS 0 bytes, default 64 [ 0.956283] Unpacking initramfs... [ 2.405442] debug: unmapping init [mem 0xffff99a23cc54000-0xffff99a23ffbffff] [ 2.409749] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.412821] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.416118] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.926789] Initialise system trusted keyrings [ 2.928971] Key type blacklist registered [ 2.931430] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.941393] zbud: loaded [ 2.944865] *** VALIDATE nfs *** [ 2.946501] *** VALIDATE nfs4 *** [ 2.949685] pstore: using deflate compression [ 2.953582] Platform Keyring initialized [ 3.064714] NET: Registered protocol family 38 [ 3.066755] Key type asymmetric registered [ 3.068468] Asymmetric key parser 'x509' registered [ 3.070833] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.074391] io scheduler mq-deadline registered [ 3.076889] io scheduler kyber registered [ 3.079135] io scheduler bfq registered [ 3.081276] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.084949] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.088733] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.092252] ACPI: Power Button [PWRF] [ 3.098345] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.105955] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.119892] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.130872] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.149467] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.177418] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.207716] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.212593] Non-volatile memory driver v1.3 [ 3.213786] Linux agpgart interface v0.103 [ 3.244988] virtio_blk virtio1: [vda] 150424 512-byte logical blocks (77.0 MB/73.4 MiB) [ 3.249503] vda: detected capacity change from 0 to 77017088 [ 3.265692] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.269268] vdb: detected capacity change from 0 to 1073741824 [ 3.283742] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.287960] vdc: detected capacity change from 0 to 2621440000 [ 3.304171] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.307869] vdd: detected capacity change from 0 to 2621440000 [ 3.322691] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.325806] vde: detected capacity change from 0 to 4294967296 [ 3.343458] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.346611] vdf: detected capacity change from 0 to 4294967296 [ 3.354080] libphy: Fixed MDIO Bus: probed [ 3.359971] usbcore: registered new interface driver usbserial_generic [ 3.362382] usbserial: USB Serial support registered for generic [ 3.364785] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.369637] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.371553] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.374234] mousedev: PS/2 mouse device common for all mice [ 3.377275] rtc_cmos 00:05: RTC can wake from S4 [ 3.381162] rtc_cmos 00:05: registered as rtc0 [ 3.383149] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.386454] intel_pstate: CPU model not supported [ 3.388553] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.389120] hid: raw HID events driver (C) Jiri Kosina [ 3.396365] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.398072] usbcore: registered new interface driver usbhid [ 3.402870] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.404071] usbhid: USB HID core driver [ 3.409317] drop_monitor: Initializing network drop monitor service [ 3.411665] Initializing XFRM netlink socket [ 3.413165] NET: Registered protocol family 10 [ 3.415561] Segment Routing with IPv6 [ 3.416927] NET: Registered protocol family 17 [ 3.418708] mpls_gso: MPLS GSO support [ 3.423820] RAS: Correctable Errors collector initialized. [ 3.425733] AVX version of gcm_enc/dec engaged. [ 3.426919] AES CTR mode by8 optimization enabled [ 3.504552] sched_clock: Marking stable (3504521494, 0)->(4401724353, -897202859) [ 3.509593] registered taskstats version 1 [ 3.512561] Loading compiled-in X.509 certificates [ 3.515467] zswap: loaded using pool lzo/zbud [ 3.540649] Key type big_key registered [ 3.553354] Key type encrypted registered [ 3.555176] ima: No TPM chip found, activating TPM-bypass! [ 3.557561] ima: Allocated hash algorithm: sha1 [ 3.559398] ima: No architecture policies found [ 3.561331] evm: Initialising EVM extended attributes: [ 3.563488] evm: security.selinux [ 3.564785] evm: security.ima [ 3.565968] evm: security.capability [ 3.567386] evm: HMAC attrs: 0x1 [ 3.570420] rtc_cmos 00:05: setting system clock to 2026-09-05 01:59:36 UTC (1788573576) [ 3.578704] debug: unmapping init [mem 0xffffffffba603000-0xffffffffba7fffff] [ 3.582028] debug: unmapping init [mem 0xffffffffb9382000-0xffffffffb9658fff] [ 3.592096] Write protecting the kernel read-only data: 28672k [ 3.595766] debug: unmapping init [mem 0xffffffffb7a03000-0xffffffffb7bfffff] [ 3.598842] debug: unmapping init [mem 0xffffffffb8314000-0xffffffffb83fffff] [ 3.641327] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.649922] systemd[1]: Detected virtualization kvm. [ 3.652154] systemd[1]: Detected architecture x86-64. [ 3.654300] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.682197] systemd[1]: No hostname configured. [ 3.684675] systemd[1]: Set hostname to . [ 3.687069] random: systemd: uninitialized urandom read (16 bytes read) [ 3.689645] systemd[1]: Initializing machine ID from random generator. [ 3.750557] random: ln: uninitialized urandom read (6 bytes read) [ 3.854793] random: systemd: uninitialized urandom read (16 bytes read) [ 3.857983] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.863482] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.867847] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Swap. [ OK ] Reached target Paths. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.510487] device-mapper: uevent: version 1.0.3 [ 4.512892] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.240233] virtio_net virtio0 ens2: renamed from eth0 [ 5.266602] random: fast init done [ 5.287641] scsi host0: ata_piix [ 5.293407] scsi host1: ata_piix [ 5.295347] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.298325] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.902185] dracut-initqueue[586]: RTNETLINK answers: File exists [ 10.073388] random: crng init done [ 10.075222] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.636843] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.834067] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.122357] SELinux: Disabled at runtime. [ 12.181432] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.191149] systemd[1]: Detected virtualization kvm. [ 12.193312] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.718164] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.722616] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.729572] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.735483] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.739746] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.748681] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.754331] systemd[1]: Created slice User and Session Slice. [ OK ] Created slice User and Session Slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on Process Core Dump Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting POSIX Message Queue File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target rpc_pipefs.target. Mounting Huge Pages File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Control Socket. Mounting Kernel Debug File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Created slice system-getty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Initrd File Systems. [ 12.871898] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Starting Apply Kernel Variables... [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ 13.153255] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.499601] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.536742] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.678547] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.692787] EDAC sbridge: Ver: 1.1.2 [ 15.210396] Key type dns_resolver registered [ 15.516587] NFS: Registering the id_resolver key type [ 15.519549] Key type id_resolver registered [ 15.520970] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started dnf makecache --timer. Starting Login Service... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg322-server login: [ 93.310547] hrtimer: interrupt took 9454242 ns [ 108.971527] libcfs: loading out-of-tree module taints kernel. [ 109.025963] Key type ._llcrypt registered [ 109.027722] Key type .llcrypt registered [ 109.129518] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 128.738794] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 130.837269] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 130.854538] alg: No test for adler32 (adler32-zlib) [ 132.293492] Lustre: Lustre: Build Version: 2.17.58_4_g27fe0b7 [ 133.304036] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 135.280765] Key type lgssc registered [ 137.766797] Lustre: Echo OBD driver; http://www.lustre.org/ [ 157.063674] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 197.162032] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 210.510615] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 210.533117] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 211.778594] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 211.825340] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 211.939423] Lustre: lustre-MDT0000: new disk, initializing [ 212.031265] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 212.050847] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 215.911860] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 228.281872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 228.434893] Lustre: 6527:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 228.501544] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 228.507147] Lustre: Skipped 1 previous similar message [ 228.625342] Lustre: lustre-MDT0001: new disk, initializing [ 228.722517] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 228.757750] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 228.767534] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 233.025953] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 237.971908] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 248.222418] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 248.527216] Lustre: lustre-OST0000: new disk, initializing [ 248.531404] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 248.538572] Lustre: 8467:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 248.618412] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 254.565393] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 255.028631] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 255.043988] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 255.101100] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 270.242963] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 270.349934] Lustre: lustre-OST0001: new disk, initializing [ 270.356437] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 270.362135] Lustre: 9540:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 270.458238] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 276.094928] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 276.108569] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 276.198158] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 277.020514] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 291.202569] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 298.544601] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 304.459844] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing check_logdir /tmp/testlogs/ [ 309.128137] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing yml_node [ 313.977616] Lustre: DEBUG MARKER: Client: 2.17.58.4 [ 316.555742] Lustre: DEBUG MARKER: MDS: 2.17.58.4 [ 319.319607] Lustre: DEBUG MARKER: OSS: 2.17.58.4 [ 321.062879] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Sep 4 22:04:50 EDT 2026 [ 341.508457] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 343.587746] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 361.963394] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 361.976253] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 361.991146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 363.492774] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 363.494806] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 363.513176] Lustre: Skipped 2 previous similar messages [ 363.538511] Lustre: Skipped 3 previous similar messages [ 368.611705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 368.620413] Lustre: Skipped 3 previous similar messages [ 373.729713] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 373.735716] Lustre: Skipped 3 previous similar messages [ 376.288186] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 376.588801] Lustre: server umount lustre-MDT0000 complete [ 380.888578] LustreError: 6522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788573953 with bad export cookie 14952450240173958161 [ 380.895052] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 380.899761] LustreError: 6522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 383.969331] LustreError: 12183:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 383.969430] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 383.972811] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 383.995985] LustreError: 12183:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 384.011401] Lustre: Skipped 1 previous similar message [ 386.017240] LustreError: 10118:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 386.029031] LustreError: 10118:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 390.114308] LustreError: 12183:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 390.129819] LustreError: 12183:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 395.234433] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 395.238723] LustreError: 6536:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 395.243766] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 395.243787] Lustre: Skipped 5 previous similar messages [ 395.282181] LustreError: 6536:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 395.523413] Lustre: server umount lustre-MDT0001 complete [ 399.651259] Lustre: server umount lustre-OST0000 complete [ 403.531345] Lustre: server umount lustre-OST0001 complete [ 410.417596] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 419.424592] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 456.938674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 457.135938] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 457.181346] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 457.287831] Lustre: lustre-MDT0000: new disk, initializing [ 457.379620] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 457.411267] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 461.697547] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 472.286297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 472.396984] Lustre: 15098:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 472.452692] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 472.460144] Lustre: Skipped 1 previous similar message [ 472.556425] Lustre: lustre-MDT0001: new disk, initializing [ 472.640113] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 472.684355] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 472.703787] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 477.499222] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 486.901282] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 492.746681] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 499.677338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 499.988225] Lustre: lustre-OST0000: new disk, initializing [ 499.998219] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 500.009520] Lustre: 17067:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 500.126340] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 501.969441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 501.978660] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 502.086037] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 506.206434] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 516.121885] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 520.896448] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 521.159863] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 525.872136] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 526.089318] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 532.967102] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 532.972098] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 532.983024] LustreError: Skipped 1 previous similar message [ 533.004070] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 533.004078] Lustre: Skipped 1 previous similar message [ 542.176415] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 542.441322] Lustre: server umount lustre-OST0000 complete [ 548.321504] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 548.349862] Lustre: Skipped 1 previous similar message [ 552.725831] Lustre: server umount lustre-MDT0000 complete [ 553.451608] LustreError: 15109:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 553.463355] LustreError: 15109:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 556.389410] LustreError: 15894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788574129 with bad export cookie 14952450240173959694 [ 556.392715] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 556.408953] LustreError: 15894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 556.800365] Lustre: server umount lustre-MDT0001 complete [ 561.889990] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 22:08:51 (1788574131) [ 564.476583] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 566.182991] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 22:08:56 (1788574136) [ 573.145685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 573.560861] LustreError: 19488:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 573.613875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 577.964524] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 586.510993] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 592.745622] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 601.567700] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 607.852801] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 614.395187] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 621.270935] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 629.547853] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 641.511782] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 641.516372] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 641.534989] Lustre: Skipped 1 previous similar message [ 641.540874] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 641.559292] Lustre: Skipped 6 previous similar messages [ 646.227592] Lustre: server umount lustre-OST0000 complete [ 657.793797] Lustre: server umount lustre-MDT0000 complete [ 661.987973] LustreError: 20591:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 661.997114] LustreError: 20591:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 662.792019] LustreError: 19469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788574235 with bad export cookie 14952450240173960576 [ 662.792781] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 662.809594] LustreError: 19469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 663.130861] Lustre: server umount lustre-MDT0001 complete [ 675.387382] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 678.436635] Key type lgssc unregistered [ 678.730167] LNet: 23020:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 678.741194] LNetError: 23020:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 678.759421] LNet: Removed LNI 192.168.203.122@tcp [ 679.682384] Key type .llcrypt unregistered [ 679.687647] Key type ._llcrypt unregistered [ 703.216996] Key type ._llcrypt registered [ 703.222679] Key type .llcrypt registered [ 703.357315] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 704.285795] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 704.356907] alg: No test for adler32 (adler32-zlib) [ 705.532720] Lustre: Lustre: Build Version: 2.17.58_4_g27fe0b7 [ 705.754429] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 707.464155] Key type lgssc registered [ 708.568721] Lustre: Echo OBD driver; http://www.lustre.org/ [ 717.036935] loop: module loaded [ 717.088090] loop0: detected capacity change from 0 to 204800000 [ 717.271912] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 717.289922] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 717.685828] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 723.250360] loop0: detected capacity change from 0 to 204800000 [ 723.416843] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 723.426161] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 723.442244] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 728.826246] loop0: detected capacity change from 0 to 204800000 [ 734.355854] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 734.389425] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 735.644539] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 735.689223] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 735.782583] Lustre: lustre-MDT0000: new disk, initializing [ 735.863739] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 735.895347] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 740.501094] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 749.257347] loop1: detected capacity change from 0 to 204800000 [ 755.489884] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 755.807737] Lustre: lustre-OST07c6: new disk, initializing [ 755.811421] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 755.815530] Lustre: Skipped 1 previous similar message [ 755.821897] Lustre: 26205:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 755.901309] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 762.727339] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 764.556602] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 764.570079] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 764.608922] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 768.558782] Lustre: 25194:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 768.581183] Lustre: 25194:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 768.590364] Lustre: 25194:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 768.600635] Lustre: 25194:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 768.605822] Lustre: 25194:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 768.611666] Lustre: 25194:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 784.267415] Lustre: 25194:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 784.276951] Lustre: 25194:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 784.282218] Lustre: 25194:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 784.290742] Lustre: 25194:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 784.296748] Lustre: 25194:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 784.302230] Lustre: 25194:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 789.661352] Lustre: 25195:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 789.678311] Lustre: 25195:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 789.692713] Lustre: 25195:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 789.704847] Lustre: 25195:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 789.717968] Lustre: 25195:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 789.736905] Lustre: 25195:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 789.748747] Lustre: 25195:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 789.757294] Lustre: 25195:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 789.765840] Lustre: 25195:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 789.773700] Lustre: 25195:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 789.784372] Lustre: 25195:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 789.794505] Lustre: 25195:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 793.023614] Lustre: 26759:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 793.030411] Lustre: 26759:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 793.038159] Lustre: 26759:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 793.049385] Lustre: 26759:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 793.060713] Lustre: 26759:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 793.066466] Lustre: 26759:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 793.075331] Lustre: 26759:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 793.086342] Lustre: 26759:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 793.100144] Lustre: 26759:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 793.107620] Lustre: 26759:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 793.117681] Lustre: 26759:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 793.129778] Lustre: 26759:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 798.066518] Lustre: 26759:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 798.078732] Lustre: 26759:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 798.090730] Lustre: 26759:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 798.103150] Lustre: 26759:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 798.113841] Lustre: 26759:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 798.125750] Lustre: 26759:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 798.137939] Lustre: 26759:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 798.154162] Lustre: 26759:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 798.163769] Lustre: 26759:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 798.180579] Lustre: 26759:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 798.197082] Lustre: 26759:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 798.203577] Lustre: 26759:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 806.349485] Lustre: 25194:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 806.360487] Lustre: 25194:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 806.367723] Lustre: 25194:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 806.378694] Lustre: 25194:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 806.384610] Lustre: 25194:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 806.391662] Lustre: 25194:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 806.396902] Lustre: 25194:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 806.402099] Lustre: 25194:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 806.410633] Lustre: 25194:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 806.418639] Lustre: 25194:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 806.429800] Lustre: 25194:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 806.440793] Lustre: 25194:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 861.686933] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 861.705633] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 861.732300] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 866.401395] Lustre: server umount lustre-OST07c6 complete [ 875.934692] Lustre: server umount lustre-MDT0000 complete [ 891.192345] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 893.671717] Key type lgssc unregistered [ 894.023546] LNet: 28763:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 894.039809] LNetError: 28763:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 894.060745] LNet: Removed LNI 192.168.203.122@tcp [ 894.904898] Key type .llcrypt unregistered [ 894.908930] Key type ._llcrypt unregistered [ 914.881372] Key type ._llcrypt registered [ 914.885699] Key type .llcrypt registered [ 915.033971] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 916.425828] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 916.517047] alg: No test for adler32 (adler32-zlib) [ 917.721490] Lustre: Lustre: Build Version: 2.17.58_4_g27fe0b7 [ 918.074932] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 919.784105] Key type lgssc registered [ 921.031869] Lustre: Echo OBD driver; http://www.lustre.org/ [ 929.118370] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 929.141459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 930.604949] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 935.071341] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 941.028905] LustreError: 30034:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 941.057271] LustreError: 30034:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 943.640805] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 944.032445] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 948.561674] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 957.464541] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 963.132762] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 969.808989] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 970.283562] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 975.412570] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 976.432441] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 984.010663] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1004.014817] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1004.031963] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1004.039573] Lustre: Skipped 1 previous similar message [ 1004.070434] Lustre: Skipped 1 previous similar message [ 1004.553371] Lustre: server umount lustre-OST0000 complete [ 1010.660603] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1010.668191] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1010.703329] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1014.242555] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1014.248911] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1015.797303] Lustre: server umount lustre-MDT0000 complete [ 1019.368763] LustreError: 30030:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1019.391242] LustreError: 30030:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1020.280614] LustreError: 30013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788574593 with bad export cookie 1359494684753230734 [ 1020.293851] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1020.876238] Lustre: server umount lustre-MDT0001 complete [ 1034.012100] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 1037.103663] Key type lgssc unregistered [ 1037.391693] LNet: 33766:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1037.404264] LNetError: 33766:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1037.424311] LNet: Removed LNI 192.168.203.122@tcp [ 1038.505104] Key type .llcrypt unregistered [ 1038.508743] Key type ._llcrypt unregistered [ 1059.926767] Key type ._llcrypt registered [ 1059.931982] Key type .llcrypt registered [ 1060.067653] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 1061.267504] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1061.291295] alg: No test for adler32 (adler32-zlib) [ 1062.461605] Lustre: Lustre: Build Version: 2.17.58_4_g27fe0b7 [ 1062.701582] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 1064.424235] Key type lgssc registered [ 1065.824805] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1076.272898] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1076.324359] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1078.442412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1084.344359] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1089.002044] LustreError: 35032:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1089.027967] LustreError: 35032:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1094.123451] LustreError: 35033:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1095.454966] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1096.043278] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1101.799304] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1110.470563] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1115.823779] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1124.893130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1125.406587] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1132.606948] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1141.670710] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1148.997921] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1159.138023] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1159.148384] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1159.154443] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1159.165288] Lustre: Skipped 1 previous similar message [ 1159.179961] Lustre: Skipped 1 previous similar message [ 1164.260141] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1164.275240] Lustre: Skipped 1 previous similar message [ 1164.805711] Lustre: server umount lustre-OST0000 complete [ 1169.384590] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1169.385740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1169.409359] Lustre: Skipped 1 previous similar message [ 1169.424402] Lustre: Skipped 1 previous similar message [ 1174.499433] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1174.502574] Lustre: Skipped 1 previous similar message [ 1174.743351] Lustre: server umount lustre-MDT0000 complete [ 1178.689337] LustreError: 35012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788574751 with bad export cookie 14018261085554129741 [ 1178.693813] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1178.705374] LustreError: 35012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1179.009137] Lustre: server umount lustre-MDT0001 complete [ 1189.762093] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 1192.416906] Key type lgssc unregistered [ 1192.672773] LNet: 38591:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1192.683342] LNetError: 38591:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1192.704296] LNet: Removed LNI 192.168.203.122@tcp [ 1193.609085] Key type .llcrypt unregistered [ 1193.613280] Key type ._llcrypt unregistered [ 1218.765726] Key type ._llcrypt registered [ 1218.768954] Key type .llcrypt registered [ 1218.888818] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 1219.934142] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1220.012465] alg: No test for adler32 (adler32-zlib) [ 1221.200835] Lustre: Lustre: Build Version: 2.17.58_4_g27fe0b7 [ 1221.463764] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 1223.144960] Key type lgssc registered [ 1224.302912] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1233.277574] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1233.318259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1234.804086] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1239.878759] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1244.136149] LustreError: 40004:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1244.157213] LustreError: 40004:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1247.828682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1248.204545] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1252.773765] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1259.861481] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1265.828734] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1272.668683] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1272.932091] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1279.098365] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1279.598585] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1287.412666] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1294.288273] Lustre: server umount lustre-MDT0000 complete [ 1294.305544] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1294.313177] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1294.329969] LustreError: 40732:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1294.821723] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1294.841392] Lustre: Skipped 1 previous similar message [ 1298.006539] LustreError: 39984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788574870 with bad export cookie 9717813377375858903 [ 1298.016797] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1298.023900] LustreError: 39984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1299.938993] LustreError: 40001:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1299.945920] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1299.957710] LustreError: 40001:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1299.989850] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1303.969808] LustreError: 40000:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1303.969929] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1303.993120] LustreError: 40000:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1304.019051] Lustre: Skipped 1 previous similar message [ 1304.552856] Lustre: server umount lustre-MDT0001 complete [ 1334.240347] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1334.425423] Lustre: server umount lustre-OST0000 complete [ 1355.216613] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 1359.494866] Key type lgssc unregistered [ 1360.076422] LNet: 43864:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1360.093432] LNetError: 43864:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1360.125277] LNet: Removed LNI 192.168.203.122@tcp [ 1361.786868] Key type .llcrypt unregistered [ 1361.794573] Key type ._llcrypt unregistered [ 1389.824078] Key type ._llcrypt registered [ 1389.828171] Key type .llcrypt registered [ 1389.951169] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 1391.248276] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1391.280767] alg: No test for adler32 (adler32-zlib) [ 1392.458741] Lustre: Lustre: Build Version: 2.17.58_4_g27fe0b7 [ 1392.746252] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 1394.416155] Key type lgssc registered [ 1395.707399] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1404.246257] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1404.259361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1405.776214] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1411.070625] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1415.142749] LustreError: 45278:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1415.156240] LustreError: 45278:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1419.948110] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1420.377909] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1426.116025] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1435.134597] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1441.453740] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1450.556344] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1451.198826] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1456.650653] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1458.245443] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1466.562151] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1474.234968] Lustre: lustre-OST0000: Not available for connect from 192.168.203.22@tcp (stopping) [ 1476.083379] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1476.112949] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1476.115249] Lustre: Skipped 1 previous similar message [ 1476.122133] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1476.135065] Lustre: Skipped 1 previous similar message [ 1479.298335] LustreError: 47102:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1479.442774] Lustre: server umount lustre-OST0000 complete [ 1501.677938] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1501.681381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1501.696497] Lustre: Skipped 1 previous similar message [ 1501.707287] Lustre: Skipped 1 previous similar message [ 1506.288800] Lustre: server umount lustre-MDT0000 complete [ 1506.785891] LustreError: 45279:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1506.819933] LustreError: 45279:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1510.621291] LustreError: 45261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788575083 with bad export cookie 15605010375459495063 [ 1510.623869] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1510.652837] LustreError: 45261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1516.957618] Lustre: server umount lustre-MDT0001 complete [ 1527.515787] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 1530.200799] Key type lgssc unregistered [ 1530.561348] LNet: 48989:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1530.575061] LNetError: 48989:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1530.612798] LNet: Removed LNI 192.168.203.122@tcp [ 1531.496558] Key type .llcrypt unregistered [ 1531.500692] Key type ._llcrypt unregistered [ 1556.013958] Key type ._llcrypt registered [ 1556.016355] Key type .llcrypt registered [ 1556.161387] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 1557.181437] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1557.233307] alg: No test for adler32 (adler32-zlib) [ 1558.325525] Lustre: Lustre: Build Version: 2.17.58_4_g27fe0b7 [ 1558.728512] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 1560.472353] Key type lgssc registered [ 1561.976518] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1570.569599] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1570.599254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1572.054623] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1576.636958] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1581.545664] LustreError: 50405:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1581.558413] LustreError: 50405:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1584.619920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1584.937320] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1590.163902] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1598.276680] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1604.135860] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1610.959285] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1611.380543] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1616.498298] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1617.738594] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1624.726502] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1633.981352] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1637.690640] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1650.055757] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-0422h26m59s [ 1651.658092] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1655.787310] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1655.797972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1655.807672] Lustre: Skipped 1 previous similar message [ 1655.821875] Lustre: Skipped 2 previous similar messages [ 1656.876361] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1656.890480] Lustre: Skipped 2 previous similar messages [ 1658.004465] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.22@tcp (stopping) [ 1660.190583] Lustre: server umount lustre-MDT0000 complete [ 1662.960677] LustreError: 51803:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1663.020051] LustreError: 51803:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1663.994666] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1665.187383] LustreError: 50401:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1669.093447] LustreError: 50405:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1669.113186] LustreError: 50405:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1671.418423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1671.779802] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1672.237478] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1672.353829] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1673.279926] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1677.624771] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1682.341930] Lustre: DEBUG MARKER: Wait for df (36701) ... [ 1684.612464] Lustre: DEBUG MARKER: done [ 1691.622125] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1691.628887] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1691.640893] Lustre: Skipped 1 previous similar message [ 1691.646464] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1691.650884] Lustre: Skipped 1 previous similar message [ 1692.644088] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1692.652578] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1695.889207] Lustre: server umount lustre-OST0000 complete [ 1703.906827] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1703.923276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1703.935424] Lustre: Skipped 4 previous similar messages [ 1705.951335] Lustre: server umount lustre-MDT0000 complete [ 1709.823236] LustreError: 51139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788575282 with bad export cookie 11178587413746131597 [ 1709.838649] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1709.848667] LustreError: 51139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1710.260239] Lustre: server umount lustre-MDT0001 complete [ 1720.134632] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 1722.524587] Key type lgssc unregistered [ 1722.800619] LNet: 55325:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1722.807782] LNetError: 55325:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1722.826975] LNet: Removed LNI 192.168.203.122@tcp [ 1723.781736] Key type .llcrypt unregistered [ 1723.784409] Key type ._llcrypt unregistered [ 1762.056996] Key type ._llcrypt registered [ 1762.059691] Key type .llcrypt registered [ 1762.155874] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 1763.417834] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1763.441778] alg: No test for adler32 (adler32-zlib) [ 1764.529778] Lustre: Lustre: Build Version: 2.17.58_4_g27fe0b7 [ 1764.744648] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 1766.432284] Key type lgssc registered [ 1767.566417] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1774.595501] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1774.612119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1775.836083] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1775.866733] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1776.005572] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1780.520973] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1788.857637] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1789.003791] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1789.033729] Lustre: 57532:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1789.289770] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1793.503224] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1800.500406] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1806.172939] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1812.944142] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1813.172881] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1813.340601] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1819.744958] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1823.369496] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1827.173873] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1834.104899] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-0422h30m03s [ 1835.559532] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1842.239981] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1844.614236] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1844.647973] LustreError: 57538:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1864.872611] Lustre: lustre-MDT0000: Client d8e524d3-30e1-4295-ac89-ca5c63096acf (at 192.168.203.22@tcp) reconnecting [ 1884.648121] LustreError: 57538:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1887.005972] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1890.173388] Lustre: DEBUG MARKER: done [ 1900.002463] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1900.009888] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1900.017917] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1900.521627] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1900.538404] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1900.539113] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1900.558716] Lustre: Skipped 1 previous similar message [ 1905.641269] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1905.652949] Lustre: Skipped 1 previous similar message [ 1910.762048] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1910.774403] Lustre: Skipped 1 previous similar message [ 1913.313429] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1913.437403] Lustre: server umount lustre-OST0000 complete [ 1917.409962] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1917.417874] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1917.449686] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1925.390485] Lustre: server umount lustre-MDT0000 complete [ 1928.874820] LustreError: 57523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788575501 with bad export cookie 16389158127176926636 [ 1928.876927] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1928.891756] LustreError: 57523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1929.258732] Lustre: server umount lustre-MDT0001 complete [ 1939.845213] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 1942.458938] Key type lgssc unregistered [ 1942.745850] LNet: 61864:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1942.762929] LNetError: 61864:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1942.783852] LNet: Removed LNI 192.168.203.122@tcp [ 1943.592894] Key type .llcrypt unregistered [ 1943.595883] Key type ._llcrypt unregistered [ 1985.035816] Key type ._llcrypt registered [ 1985.038963] Key type .llcrypt registered [ 1985.177769] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 1986.939281] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1986.978574] alg: No test for adler32 (adler32-zlib) [ 1988.226697] Lustre: Lustre: Build Version: 2.17.58_4_g27fe0b7 [ 1988.581130] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 1990.296225] Key type lgssc registered [ 1991.462591] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1994.846694] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1994.874114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1996.086677] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1996.124975] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1996.327917] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1997.409608] Lustre: Failing over lustre-MDT0000 [ 1997.608681] Lustre: server umount lustre-MDT0000 complete [ 2005.459200] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 22:32:54 (1788575574) [ 2012.661287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2013.156995] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2019.093047] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2028.068728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2028.195620] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2028.281981] Lustre: 64660:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2028.692751] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2032.991693] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2040.842746] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2047.056884] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2053.928736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2054.175341] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2054.381154] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2060.034322] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2062.002085] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2067.725692] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2073.615616] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2082.792169] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2082.798964] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2082.805110] Lustre: Skipped 1 previous similar message [ 2086.135168] Lustre: server umount lustre-MDT0000 complete [ 2089.461598] LustreError: 64651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788575662 with bad export cookie 14565195956092378387 [ 2089.466213] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2089.468783] LustreError: 64651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2089.814231] Lustre: server umount lustre-MDT0001 complete [ 2092.859987] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2096.779485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2104.102934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2108.384989] Lustre: 63582:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788575665/real 1788575665] req@00000000066c0f6b x1875457407491712/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788575681 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2108.401757] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2109.409733] Lustre: 63581:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788575666/real 1788575666] req@00000000ba9edc41 x1875457407491840/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788575682 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2114.528547] Lustre: 63580:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788575671/real 1788575671] req@00000000d92d988c x1875457407491968/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788575687 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2114.592986] LustreError: 63578:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000070a948ad x1875457407492736/t0(0) o250->MGC192.168.203.122@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2114.859112] LustreError: 66508:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2114.880140] LustreError: 66508:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2114.937915] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2115.014426] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2117.986506] LustreError: 66509:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2118.621385] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2119.021505] LustreError: 68427:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2119.054480] LustreError: 68427:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2121.056236] Lustre: 63581:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788575677/real 1788575677] req@00000000c1cfe53b x1875457407492352/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788575693 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2121.088860] Lustre: 63581:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2124.258320] LustreError: 68428:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2124.270665] LustreError: 68428:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2127.767963] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2128.162488] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2131.975308] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2133.490869] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2137.584429] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2138.350275] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2144.440388] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2150.686614] Lustre: server umount lustre-MDT0000 complete [ 2152.931513] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2152.934330] LustreError: 66509:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2152.944822] Lustre: Skipped 1 previous similar message [ 2154.242861] LustreError: 64652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788575727 with bad export cookie 14565195956092380662 [ 2154.254659] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2156.605460] Lustre: server umount lustre-MDT0001 complete [ 2162.825159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2173.408154] Lustre: 63580:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788575730/real 1788575730] req@0000000084f44886 x1875457407517568/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788575746 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2173.472353] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2173.503959] Lustre: Skipped 1 previous similar message [ 2173.618099] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2180.004207] LustreError: 66509:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2180.026063] LustreError: 66509:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2180.079361] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2180.142165] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2184.658863] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2185.184567] Lustre: 63579:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788575742/real 1788575742] req@000000002e7913e6 x1875457407518208/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788575758 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2185.210615] Lustre: 63579:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 2192.369537] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2193.798184] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2198.683655] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2199.539685] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2206.983963] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2213.136488] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2225.126366] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2225.128690] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2225.133631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2225.133641] Lustre: Skipped 2 previous similar messages [ 2225.144888] Lustre: Skipped 1 previous similar message [ 2226.573074] Lustre: server umount lustre-MDT0000 complete [ 2230.225432] LustreError: 65395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788575803 with bad export cookie 14565195956092382041 [ 2230.233319] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2230.239719] LustreError: 65395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2230.288048] LustreError: 71110:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2230.291305] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2230.307330] LustreError: 71110:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 2230.329629] Lustre: Skipped 2 previous similar messages [ 2230.571799] Lustre: server umount lustre-MDT0001 complete [ 2236.485243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2239.936514] Lustre: DEBUG MARKER: files compared the same [ 2241.833118] Lustre: server umount lustre-OST0000 complete [ 2258.195269] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 2260.674877] Key type lgssc unregistered [ 2260.948641] LNet: 74384:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2260.953788] LNetError: 74384:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2260.970862] LNet: Removed LNI 192.168.203.122@tcp [ 2261.823348] Key type .llcrypt unregistered [ 2261.828848] Key type ._llcrypt unregistered [ 2283.827440] Key type ._llcrypt registered [ 2283.829918] Key type .llcrypt registered [ 2283.930814] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 2284.881363] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2284.944240] alg: No test for adler32 (adler32-zlib) [ 2286.008627] Lustre: Lustre: Build Version: 2.17.58_4_g27fe0b7 [ 2286.261687] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 2287.928462] Key type lgssc registered [ 2288.819701] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2295.924527] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2295.943594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2297.349805] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2300.938218] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug +malloc all [ 2307.563228] LustreError: 75799:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2307.573183] LustreError: 75799:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2308.289698] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2308.505219] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2312.441346] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug +malloc all [ 2320.170933] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2326.066114] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2332.264703] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2332.576533] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2338.350037] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug +malloc all [ 2342.825494] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2344.225799] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2353.120659] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2353.125502] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2353.128262] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2353.131140] Lustre: Skipped 1 previous similar message [ 2356.754333] Lustre: server umount lustre-OST0000 complete [ 2359.776888] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2359.788297] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2359.818070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2359.826465] Lustre: Skipped 1 previous similar message [ 2363.361388] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2363.362875] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2363.390056] Lustre: Skipped 1 previous similar message [ 2365.713796] Lustre: server umount lustre-MDT0000 complete [ 2368.488821] LustreError: 76526:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2368.514869] LustreError: 76526:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2368.813834] LustreError: 75779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788575941 with bad export cookie 7153628293821436710 [ 2368.820981] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2368.826613] LustreError: 75779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2369.052663] Lustre: server umount lustre-MDT0001 complete [ 2378.241127] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 2380.709159] Key type lgssc unregistered [ 2380.996906] LNet: 79327:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2381.006409] LNetError: 79327:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2381.019186] LNet: Removed LNI 192.168.203.122@tcp [ 2381.676795] Key type .llcrypt unregistered [ 2381.679423] Key type ._llcrypt unregistered [ 2401.326450] Key type ._llcrypt registered [ 2401.328981] Key type .llcrypt registered [ 2401.399527] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 2402.033040] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2402.123913] alg: No test for adler32 (adler32-zlib) [ 2403.149518] Lustre: Lustre: Build Version: 2.17.58_4_g27fe0b7 [ 2403.335505] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 2404.992341] Key type lgssc registered [ 2405.883382] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2411.946841] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2411.960897] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2427.424530] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2432.544316] LustreError: 80692:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.122@tcp: failed processing log, type 4: rc = -110 [ 2459.232837] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2464.366790] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2470.850454] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2477.238910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2477.329831] Lustre: *** cfs_fail_loc=706, val=0*** [ 2477.591751] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2477.640610] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2480.855517] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2482.662027] LustreError: 80718:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2482.676598] LustreError: 80718:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2487.778852] LustreError: 80717:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2489.446658] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2489.791725] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2493.842070] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2501.540749] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2506.851107] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2510.305724] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2510.323519] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2510.349172] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2513.380564] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2513.386343] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2513.398890] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2514.409821] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2514.426834] Lustre: Skipped 1 previous similar message [ 2519.521611] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2519.538614] Lustre: Skipped 1 previous similar message [ 2524.130427] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2524.339438] Lustre: server umount lustre-OST0000 complete [ 2529.763686] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2529.770808] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2529.784477] Lustre: Skipped 1 previous similar message [ 2529.791116] Lustre: Skipped 1 previous similar message [ 2533.985834] Lustre: server umount lustre-MDT0000 complete [ 2534.892757] LustreError: 81782:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2534.931684] LustreError: 81782:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2537.740908] LustreError: 80700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788576110 with bad export cookie 9382171504093292701 [ 2537.745424] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2537.757033] LustreError: 80700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2538.241552] Lustre: server umount lustre-MDT0001 complete [ 2548.262391] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 2551.016639] Key type lgssc unregistered [ 2551.355466] LNet: 84308:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2551.365189] LNetError: 84308:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2551.395146] LNet: Removed LNI 192.168.203.122@tcp [ 2552.264446] Key type .llcrypt unregistered [ 2552.268053] Key type ._llcrypt unregistered [ 2575.288447] Key type ._llcrypt registered [ 2575.290974] Key type .llcrypt registered [ 2575.410637] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 2576.331961] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2576.430618] alg: No test for adler32 (adler32-zlib) [ 2577.557403] Lustre: Lustre: Build Version: 2.17.58_4_g27fe0b7 [ 2577.907264] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 2579.627254] Key type lgssc registered [ 2580.666824] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2587.768200] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2587.789293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2593.725374] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2601.356415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2601.766142] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2606.130718] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2607.078041] LustreError: 86408:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2607.095263] LustreError: 86408:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2611.172124] LustreError: 86407:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2611.188825] LustreError: 86407:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2613.685415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2613.917698] LustreError: 86408:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2613.933772] LustreError: 86408:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2613.936656] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2618.843256] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2619.363534] LustreError: 86411:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2619.388142] LustreError: 86411:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2624.486298] LustreError: 86412:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2624.505775] LustreError: 86412:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2626.134323] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2626.148834] Lustre: MGS: Client 58c889bd-b384-4e6c-abee-c2a23442bc2d (at 0@lo) reconnecting [ 2626.166094] LustreError: 86422:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2626.320109] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2626.373729] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2630.130940] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2647.009699] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2647.020406] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2647.030898] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2648.040359] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2648.045115] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2648.063844] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2650.470357] Lustre: server umount lustre-OST0000 complete [ 2658.276649] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2658.277857] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2658.279117] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2658.279123] Lustre: Skipped 1 previous similar message [ 2658.293554] Lustre: Skipped 1 previous similar message [ 2660.520603] Lustre: server umount lustre-MDT0000 complete [ 2663.406986] LustreError: 86411:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2663.423798] LustreError: 86411:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2663.915259] Lustre: server umount lustre-MDT0001 complete [ 2667.444654] Lustre: server umount MGS complete [ 2676.713270] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 22:44:06 (1788576246) [ 2706.715047] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 2719.381221] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 2750.429378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2753.799701] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2761.467876] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2761.648958] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2761.691652] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2761.749892] Lustre: lustre-MDT0001: new disk, initializing [ 2761.830830] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2761.851077] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2761.863635] Lustre: 93599:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2761.875061] LustreError: 93599:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2765.804176] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2776.220655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2776.416773] Lustre: lustre-OST0000: new disk, initializing [ 2776.420220] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2776.424949] Lustre: 94364:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2776.473476] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2782.545514] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2787.363458] Lustre: 94882:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000093c18c5f x1875458025103232/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788576376 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2790.745703] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2796.962124] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2796.972203] Lustre: MGS: Client 5f0cca14-9ba8-41f8-9b63-5d4e79df3315 (at 0@lo) reconnecting [ 2797.029591] Lustre: 95409:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2797.049572] LustreError: 93585:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2797.070518] LustreError: 93585:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2797.089981] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2797.179052] Lustre: lustre-MDT0000: new disk, initializing [ 2797.330954] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2797.364411] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2797.477979] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2798.443043] Lustre: 95434:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000000937c9ed x1875458025109632/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788576387 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2801.496103] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2807.535102] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2807.542291] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2807.618558] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2831.335394] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2831.346750] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2831.355239] Lustre: Skipped 1 previous similar message [ 2831.371135] Lustre: Skipped 2 previous similar messages [ 2835.842123] Lustre: server umount lustre-OST0000 complete [ 2841.569094] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2841.574490] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2841.588311] Lustre: Skipped 1 previous similar message [ 2841.610023] Lustre: Skipped 1 previous similar message [ 2845.783119] Lustre: server umount lustre-MDT0000 complete [ 2846.693682] LustreError: 93571:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2846.721598] LustreError: 93571:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2864.096133] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2864.454933] Lustre: server umount lustre-MDT0001 complete [ 2868.009188] Lustre: server umount MGS complete [ 2877.771213] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 22:47:27 (1788576447) [ 2894.018343] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 2896.657548] Key type lgssc unregistered [ 2896.947124] LNet: 97935:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2896.960367] LNetError: 97935:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2896.972703] LNet: Removed LNI 192.168.203.122@tcp [ 2897.699431] Key type .llcrypt unregistered [ 2897.701423] Key type ._llcrypt unregistered [ 2911.811391] Key type ._llcrypt registered [ 2911.813764] Key type .llcrypt registered [ 2911.912762] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 2912.723958] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2912.740180] alg: No test for adler32 (adler32-zlib) [ 2913.687965] Lustre: Lustre: Build Version: 2.17.58_4_g27fe0b7 [ 2913.881231] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 2915.520162] Key type lgssc registered [ 2916.471213] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2922.610195] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2922.625358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2923.999499] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2928.090746] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2933.223472] LustreError: 99133:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2933.235560] LustreError: 99133:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2935.286113] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2935.605599] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2939.875485] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2947.082643] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2952.856228] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2959.737215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2959.987356] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2965.211653] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2969.215338] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2971.946186] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2979.814193] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2979.818849] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2979.831658] Lustre: Skipped 1 previous similar message [ 2979.848073] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2984.936884] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2984.949810] Lustre: Skipped 1 previous similar message [ 2985.679126] Lustre: server umount lustre-OST0000 complete [ 2990.050746] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2990.063193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2990.068867] Lustre: Skipped 1 previous similar message [ 2990.089419] Lustre: Skipped 2 previous similar messages [ 2995.173908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2995.183718] Lustre: Skipped 1 previous similar message [ 2995.556055] Lustre: server umount lustre-MDT0000 complete [ 2998.814572] LustreError: 99115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788576571 with bad export cookie 8272078497935487909 [ 2998.817730] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2998.821484] LustreError: 99115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2999.044709] Lustre: server umount lustre-MDT0001 complete [ 3009.148324] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 3010.616504] Key type lgssc unregistered [ 3010.897759] LNet: 102665:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3010.907131] LNetError: 102665:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3010.944608] LNet: Removed LNI 192.168.203.122@tcp [ 3011.734392] Key type .llcrypt unregistered [ 3011.736265] Key type ._llcrypt unregistered [ 3026.804132] Key type ._llcrypt registered [ 3026.810737] Key type .llcrypt registered [ 3026.950898] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 3028.105413] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3028.132758] alg: No test for adler32 (adler32-zlib) [ 3029.176428] Lustre: Lustre: Build Version: 2.17.58_4_g27fe0b7 [ 3029.492059] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 3031.168179] Key type lgssc registered [ 3032.234323] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3037.694591] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3037.701979] LustreError: 103739:0:(tgt_mount.c:2573:server_fill_super()) cfs_race id 716 sleeping [ 3037.720638] LustreError: 103746:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 waking [ 3037.725107] LustreError: 103739:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 awake: rc=4984 [ 3037.734210] LustreError: 103739:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3037.744715] LustreError: 103739:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3037.746480] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3039.188720] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3046.305944] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3046.590141] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3051.101277] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3057.134230] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing lsmod [ 3060.176695] LustreError: 104951:0:(tgt_mount.c:2573:server_fill_super()) cfs_race id 716 sleeping [ 3060.183620] LustreError: 104952:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 waking [ 3060.199817] LustreError: 104951:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 awake: rc=4996 [ 3060.208815] LustreError: 104951:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3060.215625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3060.216491] LustreError: 104951:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3060.615773] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3065.828551] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3065.837595] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3065.842961] Lustre: Skipped 1 previous similar message [ 3065.852328] Lustre: Skipped 2 previous similar messages [ 3069.952230] Lustre: server umount lustre-MDT0000 complete [ 3070.945643] LustreError: 103784:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3070.961728] LustreError: 103784:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3071.977594] LustreError: 103783:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3072.002306] LustreError: 103783:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3073.730117] LustreError: 103758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788576646 with bad export cookie 9562532434310451568 [ 3073.747437] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3074.080479] LustreError: 104213:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000041a6d1c7 x1875458499039488/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3074.268524] Lustre: server umount lustre-MDT0001 complete [ 3088.161644] Lustre: server umount lustre-OST0000 complete [ 3095.754984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3096.212623] LustreError: 106016:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3096.307854] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3100.254980] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3101.683530] LustreError: 106017:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3106.790028] LustreError: 106016:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3108.512703] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3108.793519] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3112.941517] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3120.044246] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3125.627443] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3131.760153] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3132.004901] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3137.359833] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3141.256659] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3144.401703] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3150.821159] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3150.829617] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3150.845651] Lustre: Skipped 1 previous similar message [ 3150.859088] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3151.847382] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3151.850297] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3151.859782] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3151.862112] Lustre: Skipped 1 previous similar message [ 3156.214664] Lustre: server umount lustre-OST0000 complete [ 3162.085643] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3162.102977] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3166.921151] Lustre: server umount lustre-MDT0000 complete [ 3167.201108] LustreError: 106016:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3167.238835] LustreError: 106016:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3170.975815] LustreError: 105996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788576743 with bad export cookie 9562532434310452275 [ 3170.983097] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3170.986141] LustreError: 105996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 3171.316734] Lustre: server umount lustre-MDT0001 complete [ 3180.240517] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 3182.330547] Key type lgssc unregistered [ 3182.616461] LNet: 109547:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3182.622803] LNetError: 109547:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3183.663771] LNet: Removed LNI 192.168.203.122@tcp [ 3184.378349] Key type .llcrypt unregistered [ 3184.382078] Key type ._llcrypt unregistered [ 3205.078893] Key type ._llcrypt registered [ 3205.082489] Key type .llcrypt registered [ 3205.227727] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 3206.205875] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3206.335754] alg: No test for adler32 (adler32-zlib) [ 3207.391613] Lustre: Lustre: Build Version: 2.17.58_4_g27fe0b7 [ 3207.626045] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 3209.300341] Key type lgssc registered [ 3210.298524] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3216.396478] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3216.424798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3217.840727] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3221.580256] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3227.113029] LustreError: 110961:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3227.129234] LustreError: 110961:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3229.255737] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3229.616368] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3233.849248] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3240.400379] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3245.443597] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3250.003976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3250.254373] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3255.285420] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3261.622508] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3262.576087] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3270.208864] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3273.261841] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3278.309719] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3278.313922] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3278.325995] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3278.337902] Lustre: Skipped 1 previous similar message [ 3278.703961] Lustre: server umount lustre-OST0000 complete [ 3283.426065] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3283.428523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3283.442027] Lustre: Skipped 2 previous similar messages [ 3283.452333] Lustre: Skipped 1 previous similar message [ 3287.854623] Lustre: server umount lustre-MDT0000 complete [ 3288.545337] LustreError: 110961:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3288.560967] LustreError: 110961:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3291.053893] LustreError: 110941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788576863 with bad export cookie 13034629907340497966 [ 3291.058443] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3291.060968] LustreError: 110941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3291.351971] Lustre: server umount lustre-MDT0001 complete [ 3300.225179] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 3302.914333] Key type lgssc unregistered [ 3303.228722] LNet: 114595:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3303.238040] LNetError: 114595:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3304.302622] LNet: Removed LNI 192.168.203.122@tcp [ 3305.230754] Key type .llcrypt unregistered [ 3305.232515] Key type ._llcrypt unregistered [ 3324.356310] Key type ._llcrypt registered [ 3324.357980] Key type .llcrypt registered [ 3324.560761] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 3325.707836] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3325.796538] alg: No test for adler32 (adler32-zlib) [ 3326.925675] Lustre: Lustre: Build Version: 2.17.58_4_g27fe0b7 [ 3327.237766] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 3328.944517] Key type lgssc registered [ 3329.935189] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3337.269222] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3337.301815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3338.826966] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3343.192901] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3348.968343] LustreError: 115814:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3348.987741] LustreError: 115814:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3351.920387] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3352.223057] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3356.304224] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3363.818628] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3369.375410] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3375.807325] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3376.182703] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3382.267564] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3383.371193] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3390.245633] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3397.609798] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3397.615494] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3397.623937] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3398.629542] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3398.630438] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3403.703189] Lustre: server umount lustre-OST0000 complete [ 3408.867187] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3408.894856] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3408.902663] Lustre: Skipped 2 previous similar messages [ 3413.987517] LustreError: 117248:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3413.997441] LustreError: 117248:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3415.380539] Lustre: server umount lustre-MDT0000 complete [ 3419.107438] LustreError: 115810:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3419.134056] LustreError: 115810:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3419.305341] LustreError: 116549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788576992 with bad export cookie 8171247751166093574 [ 3419.307714] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3419.316129] LustreError: 116549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3419.944475] Lustre: server umount lustre-MDT0001 complete [ 3431.319150] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 3433.976726] Key type lgssc unregistered [ 3434.300885] LNet: 119347:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3434.309161] LNetError: 119347:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3434.321852] LNet: Removed LNI 192.168.203.122@tcp [ 3435.160900] Key type .llcrypt unregistered [ 3435.163889] Key type ._llcrypt unregistered [ 3457.193942] Key type ._llcrypt registered [ 3457.197386] Key type .llcrypt registered [ 3457.304808] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 3458.062455] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3458.178753] alg: No test for adler32 (adler32-zlib) [ 3459.182947] Lustre: Lustre: Build Version: 2.17.58_4_g27fe0b7 [ 3459.375663] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 3461.016171] Key type lgssc registered [ 3462.045781] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3468.160469] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3468.181682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3469.602805] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3473.696312] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3480.036748] LustreError: 120760:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3480.065762] LustreError: 120760:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3482.908278] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3483.337996] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3488.073381] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3496.889887] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3504.483290] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3511.418533] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3511.695512] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3514.832879] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3518.584857] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3527.994566] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3536.230962] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3539.518931] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3539.528187] Lustre: Skipped 2 previous similar messages [ 3543.539977] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3543.544294] Lustre: Skipped 2 previous similar messages [ 3545.680661] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3545.710860] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3556.903346] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.22@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3556.910741] Lustre: Skipped 2 previous similar messages [ 3562.573893] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.22@tcp 192.168.0.[2,10]@tcp [ 3566.666317] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.22/24@tcp in log lustre-MDT0000 [ 3566.674764] Lustre: Skipped 2 previous similar messages [ 3573.832824] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.22/24@tcp [ 3573.841803] Lustre: Skipped 1 previous similar message [ 3580.903319] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3580.911324] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3580.925403] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3582.440508] Lustre: server umount lustre-OST0000 complete [ 3587.564041] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3587.585603] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3592.675208] LustreError: 120760:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3592.693905] LustreError: 120760:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3592.724090] Lustre: server umount lustre-MDT0000 complete [ 3596.843090] LustreError: 120741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788577169 with bad export cookie 8460663246745526867 [ 3596.849837] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3596.856991] LustreError: 120741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3607.208816] Lustre: server umount lustre-MDT0001 complete [ 3617.380583] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 3619.909939] Key type lgssc unregistered [ 3620.197540] LNet: 125694:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3620.206453] LNetError: 125694:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3621.227214] LNet: Removed LNI 192.168.203.122@tcp [ 3622.030361] Key type .llcrypt unregistered [ 3622.032859] Key type ._llcrypt unregistered [ 3653.428217] Key type ._llcrypt registered [ 3653.431089] Key type .llcrypt registered [ 3653.527787] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 3654.319416] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3654.331163] alg: No test for adler32 (adler32-zlib) [ 3655.384741] Lustre: Lustre: Build Version: 2.17.58_4_g27fe0b7 [ 3655.671135] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 3657.400223] Key type lgssc registered [ 3658.275504] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3664.249252] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3664.263774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3665.601172] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3665.630118] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.22/24@tcp [ 3665.636554] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3669.653924] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3675.124118] LustreError: 127254:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3675.139323] LustreError: 127254:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3676.919773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3677.144713] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3677.165794] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.22/24@tcp [ 3677.212788] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3680.390442] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3686.732104] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3691.999629] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3698.679634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3699.050601] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3704.677414] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3710.338726] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3712.829990] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3722.721473] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3722.729758] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3722.739321] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3725.802213] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3725.810089] Lustre: Skipped 2 previous similar messages [ 3729.071866] Lustre: server umount lustre-OST0000 complete [ 3733.473169] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3733.489098] LustreError: Skipped 1 previous similar message [ 3733.496331] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3733.528045] Lustre: Skipped 1 previous similar message [ 3733.533103] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3736.035121] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3736.047866] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3741.154192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3741.169399] Lustre: Skipped 2 previous similar messages [ 3747.296163] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3747.582325] Lustre: server umount lustre-MDT0000 complete [ 3750.370780] LustreError: 127254:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3750.413820] LustreError: 127254:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3751.327327] LustreError: 127234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788577324 with bad export cookie 2519382782338498088 [ 3751.341494] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3751.344625] LustreError: 127234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3751.677614] Lustre: server umount lustre-MDT0001 complete [ 3761.563762] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 3764.394343] Key type lgssc unregistered [ 3764.867890] LNet: 130855:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3764.884956] LNetError: 130855:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3764.925471] LNet: Removed LNI 192.168.203.122@tcp [ 3766.175661] Key type .llcrypt unregistered [ 3766.180167] Key type ._llcrypt unregistered [ 3797.293472] Key type ._llcrypt registered [ 3797.299717] Key type .llcrypt registered [ 3797.538705] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 3813.658574] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 3814.571881] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3814.641035] alg: No test for adler32 (adler32-zlib) [ 3815.775910] Lustre: Lustre: Build Version: 2.17.58_4_g27fe0b7 [ 3816.083205] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 3817.768391] Key type lgssc registered [ 3819.135627] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3853.258966] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3853.297149] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3854.510040] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3854.544617] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3854.615493] Lustre: lustre-MDT0000: new disk, initializing [ 3854.687820] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3854.704088] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3858.711722] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3867.966306] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3868.068702] Lustre: 134254:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3868.115770] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3868.119371] Lustre: Skipped 1 previous similar message [ 3868.189816] Lustre: lustre-MDT0001: new disk, initializing [ 3868.285937] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3868.324867] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3868.330457] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3872.269436] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3883.690199] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3892.533286] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3902.965360] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3903.326203] Lustre: lustre-OST0000: new disk, initializing [ 3903.344635] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3903.361908] Lustre: 136222:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3903.544284] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3910.029103] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3911.776900] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3911.800769] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3911.904778] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3920.245832] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3925.128817] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3925.333307] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3930.351855] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3930.667544] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3937.765359] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3937.778208] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3937.796060] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3937.812799] Lustre: Skipped 1 previous similar message [ 3942.883605] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3942.894723] Lustre: Skipped 1 previous similar message [ 3946.976215] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3947.380528] Lustre: server umount lustre-OST0000 complete [ 3953.121951] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3953.147697] Lustre: Skipped 1 previous similar message [ 3953.163544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3953.175437] Lustre: Skipped 1 previous similar message [ 3958.241395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3958.269258] Lustre: Skipped 1 previous similar message [ 3958.822843] Lustre: server umount lustre-MDT0000 complete [ 3963.743427] LustreError: 134247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788577536 with bad export cookie 18193689632172885742 [ 3963.758039] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3963.762706] LustreError: 134247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3964.341963] Lustre: server umount lustre-MDT0001 complete [ 3973.264899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3973.915743] LustreError: 138442:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3973.949454] LustreError: 138442:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3973.994511] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3978.084572] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3979.234776] LustreError: 138443:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3983.333991] LustreError: 138442:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3986.060215] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3986.425603] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3990.363357] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3997.638327] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4003.085950] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4009.992875] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4010.240337] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4016.198966] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4023.805521] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4028.401108] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4028.636609] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4033.520708] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4033.687985] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4041.643255] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4041.822621] Lustre: lustre-OST0001: new disk, initializing [ 4041.827716] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4041.834244] Lustre: 141816:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4041.910612] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4047.800824] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4048.970603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4048.988939] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4049.096126] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4055.662959] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4055.911902] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4060.092531] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4060.360648] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4066.394800] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff977102f81000.ost_server_uuid 50 [ 4068.080621] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff977102f81000.ost_server_uuid in FULL state after 0 sec [ 4080.099616] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4080.100506] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4080.107420] LustreError: Skipped 2 previous similar messages [ 4080.120792] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4080.129530] Lustre: Skipped 2 previous similar messages [ 4080.153052] Lustre: Skipped 1 previous similar message [ 4084.527640] Lustre: server umount lustre-OST0001 complete [ 4085.234179] LustreError: 142250:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4085.260066] LustreError: 142250:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4090.343794] LustreError: 140267:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4090.346518] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4090.351452] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4090.352526] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4090.360626] LustreError: 140267:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4100.592699] LustreError: 140267:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4100.616643] LustreError: 140267:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4103.136107] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4103.259290] Lustre: server umount lustre-OST0000 complete [ 4109.281481] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4109.287487] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4109.294858] Lustre: Skipped 1 previous similar message [ 4109.298831] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4109.306171] Lustre: Skipped 5 previous similar messages [ 4113.444630] Lustre: server umount lustre-MDT0000 complete [ 4117.016133] LustreError: 138424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788577689 with bad export cookie 18193689632172886631 [ 4117.018480] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4117.025059] LustreError: 138424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4117.348599] Lustre: server umount lustre-MDT0001 complete [ 4133.741337] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 4135.926040] Key type lgssc unregistered [ 4136.206605] LNet: 144818:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4136.216971] LNetError: 144818:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4136.228275] LNet: Removed LNI 192.168.203.122@tcp [ 4136.847312] Key type .llcrypt unregistered [ 4136.849642] Key type ._llcrypt unregistered [ 4172.231407] Key type ._llcrypt registered [ 4172.233062] Key type .llcrypt registered [ 4172.332333] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 4186.363264] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 4187.901280] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4188.009533] alg: No test for adler32 (adler32-zlib) [ 4189.134937] Lustre: Lustre: Build Version: 2.17.58_4_g27fe0b7 [ 4189.399026] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 4191.064248] Key type lgssc registered [ 4192.265675] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4230.075779] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4230.120622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4231.359028] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4231.415270] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4231.522302] Lustre: lustre-MDT0000: new disk, initializing [ 4231.597704] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4231.621825] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4236.530843] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4246.946045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4247.093966] Lustre: 148963:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4247.160493] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4247.171537] Lustre: Skipped 1 previous similar message [ 4247.282889] Lustre: lustre-MDT0001: new disk, initializing [ 4247.447862] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4247.541695] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4247.551485] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4251.580928] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4260.699691] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4266.309996] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4273.043752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4273.361943] Lustre: lustre-OST0000: new disk, initializing [ 4273.370485] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4273.383834] Lustre: 150931:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4273.504350] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4279.463337] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4280.420787] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4280.432680] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4280.649528] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4291.474270] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4297.005306] Lustre: Failing over lustre-OST0000 [ 4297.226756] Lustre: server umount lustre-OST0000 complete [ 4314.094071] Lustre: 147186:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788577871/real 1788577871] req@00000000c43fc703 x1875459715400960/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1788577887 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4314.128199] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4316.158266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4316.564521] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4316.600430] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4316.620272] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4317.600370] Lustre: 147185:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788577874/real 1788577874] req@00000000143d6ce1 x1875459715402112/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788577890 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4319.593736] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4322.272096] Lustre: 147187:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788577879/real 1788577879] req@000000007d9a6585 x1875459715403264/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788577895 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4322.309866] Lustre: 147187:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4323.856714] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4325.055773] Lustre: lustre-OST0000: Recovery over after 0:06, of 3 clients 3 recovered and 0 were evicted. [ 4325.056858] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4327.392120] Lustre: 147185:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788577884/real 1788577884] req@0000000009437bb5 x1875459715404160/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788577900 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4327.417794] Lustre: 147185:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4328.376176] Lustre: Failing over lustre-MDT0000 [ 4329.442474] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4329.451359] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4330.724831] Lustre: server umount lustre-MDT0000 complete [ 4335.076447] LustreError: 148968:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4335.110314] LustreError: 148968:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 4340.193991] LustreError: 148973:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4340.225158] LustreError: 148973:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4345.313284] Lustre: 147185:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788577902/real 1788577902] req@00000000be76a309 x1875459715414016/t0(0) o400->MGC192.168.203.122@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788577918 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4345.315329] LustreError: 148974:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4345.342897] Lustre: 147185:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4345.343124] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4345.390536] LustreError: 148974:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4349.864996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4350.433613] LustreError: 151333:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4350.459468] LustreError: 151333:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4355.555537] LustreError: 148969:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4355.598258] LustreError: 148969:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4355.724178] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.22@tcp (not set up) [ 4355.857181] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4355.929272] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4359.362724] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4360.830542] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4361.203759] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4361.217462] Lustre: Skipped 1 previous similar message [ 4361.236973] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4361.270426] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4371.425590] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4371.434235] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4371.455339] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4371.512346] Lustre: Skipped 3 previous similar messages [ 4373.932632] Lustre: server umount lustre-OST0000 complete [ 4381.665350] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4381.669760] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4381.681906] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4381.681913] Lustre: Skipped 1 previous similar message [ 4381.724085] Lustre: Skipped 1 previous similar message [ 4384.913531] Lustre: server umount lustre-MDT0000 complete [ 4386.785539] LustreError: 149888:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4386.831565] LustreError: 149888:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4388.505289] LustreError: 148953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788577961 with bad export cookie 9851154409582395319 [ 4388.528847] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4388.981509] Lustre: server umount lustre-MDT0001 complete [ 4400.340890] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 4403.157230] Key type lgssc unregistered [ 4403.376202] LNet: 154587:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4403.388851] LNetError: 154587:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4403.430408] LNet: Removed LNI 192.168.203.122@tcp [ 4404.478702] Key type .llcrypt unregistered [ 4404.483255] Key type ._llcrypt unregistered [ 4441.514586] Key type ._llcrypt registered [ 4441.517646] Key type .llcrypt registered [ 4441.626658] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 4442.499503] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4442.565355] alg: No test for adler32 (adler32-zlib) [ 4443.666075] Lustre: Lustre: Build Version: 2.17.58_4_g27fe0b7 [ 4443.885533] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 4445.576132] Key type lgssc registered [ 4446.646167] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4453.966849] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4454.005630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4455.829220] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4461.277480] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4465.130368] LustreError: 156101:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4465.162802] LustreError: 156101:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4470.244469] LustreError: 156102:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4472.229723] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4475.374349] LustreError: 156101:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4476.289521] LustreError: 156131:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4476.296512] LustreError: 156131:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4476.303069] LustreError: 156131:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 21, retries 0, failed: rc = -5 [ 4476.552418] Lustre: server umount lustre-MDT0000 complete [ 4493.238237] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 4505.048554] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 4541.913303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4542.122698] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4542.148978] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4542.225959] Lustre: lustre-MDT0000: new disk, initializing [ 4542.306232] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4542.323987] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4546.499776] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4556.846265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4556.996817] Lustre: 159439:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4557.074332] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4557.082483] Lustre: Skipped 1 previous similar message [ 4557.167253] Lustre: lustre-MDT0001: new disk, initializing [ 4557.252529] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4557.300424] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4557.309057] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4561.711405] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4571.010038] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4578.391404] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4587.158832] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4587.579992] Lustre: lustre-OST0000: new disk, initializing [ 4587.588592] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4587.602079] Lustre: 161408:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4587.715664] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4593.705598] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4593.718534] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4593.847555] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4595.144948] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4603.771222] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4607.280453] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4608.754983] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 23:16:18 (1788578178) [ 4859.362248] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4859.366491] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4859.379501] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4860.385259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4860.386408] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4865.509450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4865.521508] Lustre: Skipped 4 previous similar messages [ 4870.113052] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4870.361361] Lustre: server umount lustre-MDT0000 complete [ 4870.627628] LustreError: 159447:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4870.641178] LustreError: 159447:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4873.765374] LustreError: 159432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788578446 with bad export cookie 7298723714928794785 [ 4873.772962] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4873.780845] LustreError: 159432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4875.744518] LustreError: 163185:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4875.744928] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4875.771257] Lustre: Skipped 1 previous similar message [ 4875.776962] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4880.132208] Lustre: server umount lustre-MDT0001 complete [ 4898.272280] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4898.488363] Lustre: server umount lustre-OST0000 complete [ 4909.295663] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 4921.670097] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 4955.503989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4955.696711] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4955.734920] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4955.794921] Lustre: lustre-MDT0000: new disk, initializing [ 4955.880489] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4955.914078] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4959.503476] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4969.648431] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4969.799899] Lustre: 166074:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4969.902908] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4969.918616] Lustre: Skipped 1 previous similar message [ 4970.040955] Lustre: lustre-MDT0001: new disk, initializing [ 4970.188825] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4970.256918] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4970.272762] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4974.196740] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4982.733802] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4988.098477] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4994.196259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4994.470539] Lustre: lustre-OST0000: new disk, initializing [ 4994.481417] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4994.491633] Lustre: 168039:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4994.580406] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5000.110293] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5002.805390] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5002.830266] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5003.007525] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5008.350697] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5012.664771] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5012.855613] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5016.973545] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5017.167537] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5023.716579] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5023.717906] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5023.718168] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5023.734181] Lustre: Skipped 1 previous similar message [ 5023.753952] Lustre: Skipped 2 previous similar messages [ 5024.881191] Lustre: server umount lustre-OST0000 complete [ 5028.834930] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5028.850815] Lustre: Skipped 1 previous similar message [ 5033.954469] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5033.959459] Lustre: Skipped 3 previous similar messages [ 5034.401772] Lustre: server umount lustre-MDT0000 complete [ 5038.266215] LustreError: 169580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788578611 with bad export cookie 7298723714928953608 [ 5038.273608] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5038.275940] LustreError: 169580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5038.654131] Lustre: server umount lustre-MDT0001 complete [ 5047.049994] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 23:23:36 (1788578616) [ 5061.178114] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 5069.809937] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 5099.101179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5099.317777] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5099.347925] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5099.441449] Lustre: lustre-MDT0000: new disk, initializing [ 5099.503285] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5099.528145] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5104.077139] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5113.804126] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5113.906578] Lustre: 172794:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5114.186031] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5117.815623] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5125.897570] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5130.798501] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5136.240148] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5136.420360] Lustre: lustre-OST0000: new disk, initializing [ 5136.424421] Lustre: Skipped 1 previous similar message [ 5136.428361] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5136.432410] Lustre: Skipped 2 previous similar messages [ 5136.439704] Lustre: 174769:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5141.542242] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5143.082664] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5143.095822] Lustre: Skipped 1 previous similar message [ 5143.100309] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5143.186814] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5149.536624] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5157.954505] Lustre: server umount lustre-OST0000 complete [ 5164.001514] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5164.015212] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5164.020404] Lustre: Skipped 1 previous similar message [ 5174.176162] Lustre: 155413:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788578731/real 1788578731] req@00000000f2c9ef69 x1875459982473088/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788578747 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5175.267748] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5175.487963] Lustre: server umount lustre-MDT0000 complete [ 5179.370215] LustreError: 172806:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5179.371566] Lustre: 155414:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788578736/real 1788578736] req@0000000044d3acfb x1875459982475520/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788578752 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5179.418570] LustreError: 172806:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5180.290900] LustreError: 172787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788578753 with bad export cookie 7298723714928954490 [ 5180.297200] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5180.301298] LustreError: 172787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5180.755529] Lustre: server umount lustre-MDT0001 complete [ 5192.973204] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 23:26:02 (1788578762) [ 5207.982810] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 5216.718636] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 5246.525908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5246.755624] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5246.787148] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5246.860779] Lustre: lustre-MDT0000: new disk, initializing [ 5246.925691] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5246.930765] Lustre: Skipped 2 previous similar messages [ 5246.948664] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5250.414419] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5260.006027] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5260.147674] Lustre: 179094:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5260.387358] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5264.488599] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5273.052426] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5278.581440] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5284.216481] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5284.447362] Lustre: 181070:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5289.692664] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5292.164391] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5297.643603] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5302.753488] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5302.763387] LustreError: Skipped 1 previous similar message [ 5302.767862] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5302.778349] Lustre: Skipped 2 previous similar messages [ 5302.782098] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5302.784068] Lustre: Skipped 6 previous similar messages [ 5307.366689] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5308.908937] Lustre: server umount lustre-OST0000 complete [ 5318.348294] Lustre: server umount lustre-MDT0000 complete [ 5322.213151] LustreError: 179087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788578895 with bad export cookie 7298723714928955582 [ 5322.217892] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5322.224349] LustreError: 179087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5322.489395] Lustre: server umount lustre-MDT0001 complete [ 5332.342443] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 5334.503790] Key type lgssc unregistered [ 5334.792784] LNet: 182925:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5334.800105] LNetError: 182925:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5334.819532] LNet: Removed LNI 192.168.203.122@tcp [ 5335.551426] Key type .llcrypt unregistered [ 5335.554991] Key type ._llcrypt unregistered [ 5355.306985] Key type ._llcrypt registered [ 5355.318947] Key type .llcrypt registered [ 5355.398266] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 5356.408079] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5356.484927] alg: No test for adler32 (adler32-zlib) [ 5357.626336] Lustre: Lustre: Build Version: 2.17.58_4_g27fe0b7 [ 5357.860144] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 5359.536227] Key type lgssc registered [ 5360.599371] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5366.632215] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5366.655933] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5367.945746] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5370.941587] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5377.515387] LustreError: 184316:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5377.536330] LustreError: 184316:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5377.835751] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5378.092456] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5381.969786] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5389.151494] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5395.174964] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5401.878815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5402.422601] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5409.286817] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5412.692749] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5416.700273] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5434.339372] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5434.355258] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5434.379809] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5438.436570] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5438.439335] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5438.447948] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5440.047654] Lustre: server umount lustre-OST0000 complete [ 5443.555444] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5443.565811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5443.569686] Lustre: Skipped 1 previous similar message [ 5448.609640] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5448.620102] Lustre: Skipped 4 previous similar messages [ 5449.886411] Lustre: server umount lustre-MDT0000 complete [ 5452.813086] LustreError: 184297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788579025 with bad export cookie 9123403424697524132 [ 5452.818662] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5452.830710] LustreError: 184297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5453.194752] Lustre: server umount lustre-MDT0001 complete [ 5462.839756] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 5465.413873] Key type lgssc unregistered [ 5465.700831] LNet: 187868:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5465.710458] LNetError: 187868:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5465.727909] LNet: Removed LNI 192.168.203.122@tcp [ 5466.562476] Key type .llcrypt unregistered [ 5466.565369] Key type ._llcrypt unregistered [ 5487.292882] Key type ._llcrypt registered [ 5487.298262] Key type .llcrypt registered [ 5487.406908] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 5488.086964] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5488.161271] alg: No test for adler32 (adler32-zlib) [ 5489.250778] Lustre: Lustre: Build Version: 2.17.58_4_g27fe0b7 [ 5489.507616] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 5491.208186] Key type lgssc registered [ 5492.045063] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5497.946797] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5497.963949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5499.301605] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5502.519688] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5509.540233] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5509.602423] LustreError: 189258:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5509.628327] LustreError: 189258:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5509.829206] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5513.781391] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5519.729860] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5524.543575] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5529.777019] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5530.099574] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5535.502941] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5542.015748] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5544.443816] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5553.835155] Lustre: lustre-OST0000: Not available for connect from 192.168.203.22@tcp (stopping) [ 5554.658550] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5554.669497] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5554.688238] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5554.697416] Lustre: Skipped 1 previous similar message [ 5558.308661] Lustre: server umount lustre-OST0000 complete [ 5565.081518] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff977120889800.ost_server_uuid 50 [ 5566.824691] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5581.796788] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5581.806826] LustreError: Skipped 1 previous similar message [ 5581.810739] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5581.821872] Lustre: Skipped 1 previous similar message [ 5581.829571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5584.356810] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5584.358137] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5584.366815] Lustre: Skipped 1 previous similar message [ 5586.865044] Lustre: server umount lustre-MDT0000 complete [ 5589.474471] LustreError: 189255:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5589.490811] LustreError: 189255:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5590.209198] LustreError: 189241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788579163 with bad export cookie 8689989345863291637 [ 5590.213745] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5590.219918] LustreError: 189241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5590.533767] Lustre: server umount lustre-MDT0001 complete [ 5599.305505] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 23:32:49 (1788579169) [ 5605.279226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5605.633514] LustreError: 193179:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5605.687613] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5609.651550] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5610.980887] LustreError: 193180:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5615.075137] LustreError: 193179:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5617.691804] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5618.209618] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5622.509478] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5630.174114] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5635.586465] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5641.854359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5642.126609] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5648.062861] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5650.353728] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5655.461060] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5662.659997] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5662.906987] Lustre: lustre-OST0001: new disk, initializing [ 5662.909914] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5662.918219] Lustre: 196022:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5663.022850] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5670.073876] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5672.073436] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5672.083758] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5672.231064] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5680.227142] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5684.197932] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5684.206986] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5684.225535] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5686.245924] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5689.061350] Lustre: server umount lustre-OST0000 complete [ 5691.007863] LustreError: 195474:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5695.868818] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5696.112698] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5700.583104] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5700.750913] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5708.257690] LustreError: 195474:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5708.277041] LustreError: 195474:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 5718.498381] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5718.516649] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5718.532586] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5718.535363] Lustre: Skipped 2 previous similar messages [ 5718.555865] Lustre: Skipped 5 previous similar messages [ 5720.852620] Lustre: server umount lustre-OST0001 complete [ 5728.739236] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5728.756886] Lustre: Skipped 1 previous similar message [ 5730.491592] Lustre: server umount lustre-MDT0000 complete [ 5733.526105] LustreError: 193160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788579306 with bad export cookie 8689989345863292708 [ 5733.536038] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5733.541317] LustreError: 193160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5733.812698] Lustre: server umount lustre-MDT0001 complete [ 5754.773870] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 23:35:25 (1788579325) [ 5760.046323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5760.333267] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5760.374494] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5760.551644] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5764.180576] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5771.209148] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5771.292884] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5771.318561] Lustre: 199457:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5775.701425] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5782.881563] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5788.407898] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5794.763691] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5795.059449] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5800.599304] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 5801.316462] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5808.900336] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5816.707351] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5816.761564] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5822.703529] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 5823.005994] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5829.826487] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5833.214785] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5833.228365] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5833.241558] Lustre: Skipped 4 previous similar messages [ 5838.039789] Lustre: server umount lustre-OST0000 complete [ 5838.314727] LustreError: 201306:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5838.327089] LustreError: 201306:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 5853.669502] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5853.686098] LustreError: Skipped 3 previous similar messages [ 5858.635584] Lustre: server umount lustre-OST0001 complete [ 5868.071959] Lustre: server umount lustre-MDT0000 complete [ 5871.698746] LustreError: 199450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788579444 with bad export cookie 8689989345863294115 [ 5871.708430] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5871.713463] LustreError: 199450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5872.110614] Lustre: server umount lustre-MDT0001 complete [ 5895.016696] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 23:37:45 (1788579465) [ 5910.094679] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 5920.967988] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 5956.341911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5956.556993] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5956.583529] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5956.672314] Lustre: lustre-MDT0000: new disk, initializing [ 5956.753314] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5956.761127] Lustre: Skipped 3 previous similar messages [ 5956.775489] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5960.896361] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5970.591592] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5970.704501] Lustre: 207554:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5970.748894] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5970.752926] Lustre: Skipped 1 previous similar message [ 5970.809299] Lustre: lustre-MDT0001: new disk, initializing [ 5970.933894] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5970.945885] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5975.061241] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5984.547807] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5990.434920] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5997.430738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5997.765416] Lustre: lustre-OST0000: new disk, initializing [ 5997.768185] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5997.790297] Lustre: 209522:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6003.725630] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6004.280955] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6004.298064] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6004.424981] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6013.730418] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6018.545898] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6018.746884] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6023.356969] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6023.577533] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6030.306634] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6030.309659] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6030.313489] LustreError: Skipped 2 previous similar messages [ 6030.319776] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6030.337200] Lustre: Skipped 6 previous similar messages [ 6030.341721] Lustre: Skipped 5 previous similar messages [ 6031.531402] Lustre: server umount lustre-OST0000 complete [ 6049.249145] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6049.387835] Lustre: server umount lustre-MDT0000 complete [ 6050.786126] LustreError: 210861:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6050.791791] LustreError: 210861:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 6053.298734] LustreError: 207547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788579626 with bad export cookie 8689989345863295893 [ 6053.300258] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6053.309481] LustreError: 207547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6053.620637] Lustre: server umount lustre-MDT0001 complete [ 6061.774291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6066.969784] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6076.646589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6082.109973] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6092.523584] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6099.191503] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6105.692265] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6106.047802] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6106.056414] Lustre: Skipped 4 previous similar messages [ 6112.122814] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6119.364131] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6123.321773] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6123.486731] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6127.200391] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6127.433639] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6131.687713] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6131.689300] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6131.698686] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6131.717025] Lustre: Skipped 3 previous similar messages [ 6131.732227] Lustre: Skipped 8 previous similar messages [ 6135.397231] Lustre: server umount lustre-OST0000 complete [ 6141.318308] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6146.286954] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6174.922729] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6181.154692] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6188.719909] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6193.471838] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 3 sec [ 6196.782636] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6196.801659] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6196.804961] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6201.826971] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6203.551151] Lustre: server umount lustre-OST0000 complete [ 6213.352501] Lustre: server umount lustre-MDT0000 complete [ 6216.865695] LustreError: 211723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788579789 with bad export cookie 8689989345863296782 [ 6216.866383] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6216.889247] LustreError: 211723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6217.373744] Lustre: server umount lustre-MDT0001 complete [ 6226.869505] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 23:43:16 (1788579796) [ 6233.009338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6233.451875] LustreError: 217710:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6233.473513] LustreError: 217710:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6237.141444] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6244.943861] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6249.293816] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6256.045115] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6262.088542] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6267.937724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6273.435527] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6279.682646] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6284.012365] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6284.247433] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6287.918442] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6288.105406] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6293.579078] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6293.727997] Lustre: lustre-OST0001: new disk, initializing [ 6293.732983] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6293.737967] Lustre: 221078:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6298.502381] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6301.758125] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6301.769947] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6301.865550] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6306.463890] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6310.595151] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6310.810649] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6315.021473] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6315.218760] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6317.537528] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6317.561846] Lustre: Skipped 5 previous similar messages [ 6317.565598] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6317.573219] Lustre: Skipped 6 previous similar messages [ 6331.361504] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6331.520131] Lustre: server umount lustre-OST0001 complete [ 6337.811541] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6337.993633] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6341.741883] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6341.919571] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6370.236171] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6370.412847] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6370.417673] Lustre: Skipped 5 previous similar messages [ 6375.924252] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6375.937775] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6375.937802] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6375.943674] Lustre: Skipped 1 previous similar message [ 6377.527721] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6385.777912] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6387.527712] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6391.276130] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6391.279804] LustreError: Skipped 3 previous similar messages [ 6395.639388] Lustre: server umount lustre-OST0001 complete [ 6417.809834] Lustre: server umount lustre-OST0000 complete [ 6427.484464] Lustre: server umount lustre-MDT0000 complete [ 6430.727568] LustreError: 219553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788580003 with bad export cookie 8689989345863297748 [ 6430.735070] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6430.743833] LustreError: 219553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6437.141386] Lustre: server umount lustre-MDT0001 complete [ 6460.168662] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 23:47:09 (1788580029) [ 6469.183625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6469.616477] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6469.661918] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6475.579517] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6485.242704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6485.347496] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6485.371703] Lustre: 226325:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6491.019831] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6499.721277] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6505.653599] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6511.686935] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6511.941365] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6518.250384] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6526.439860] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6535.557131] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6535.657926] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6541.930316] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6548.579964] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6549.912272] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6554.998984] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6555.229477] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6559.798446] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6559.994684] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6564.538130] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff977106d2d800.ost_server_uuid 50 [ 6565.716382] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff977106d2d800.ost_server_uuid in IDLE state after 0 sec [ 6566.583257] Lustre: Permanently deactivating lustre-OST0001 [ 6566.596914] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6569.665153] Lustre: Permanently reactivating lustre-OST0001 [ 6573.538413] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6573.555860] Lustre: Skipped 7 previous similar messages [ 6578.659677] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6578.664600] Lustre: Skipped 18 previous similar messages [ 6579.478289] Lustre: server umount lustre-OST0001 complete [ 6583.780219] LustreError: 229589:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6583.801963] LustreError: 229589:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 35 previous similar messages [ 6589.394498] Lustre: server umount lustre-OST0000 complete [ 6598.892847] Lustre: server umount lustre-MDT0000 complete [ 6602.662386] LustreError: 226317:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788580175 with bad export cookie 8689989345863299036 [ 6602.669304] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6602.674197] LustreError: 226317:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6602.975417] Lustre: server umount lustre-MDT0001 complete [ 6624.510357] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 23:49:54 (1788580194) [ 6630.777660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6631.052977] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6631.088610] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6631.108829] Lustre: Skipped 5 previous similar messages [ 6634.916788] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6642.878500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6643.008170] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6643.042636] Lustre: 233018:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6648.861527] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6656.705967] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6662.687799] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6668.899015] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6669.094533] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6669.124269] Lustre: Permanently deactivating lustre-OST0000 [ 6669.134766] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6674.764636] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6679.590879] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6682.032191] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6688.684389] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6688.758250] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6689.866410] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6694.297024] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6701.814819] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6703.702086] Lustre: Permanently reactivating lustre-OST0000 [ 6703.727150] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6703.735778] Lustre: Skipped 2 previous similar messages [ 6711.860210] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 32 seconds. I think it's dead, and I am evicting it. exp 000000006cd069cd, cur 1788580284 deadline 1788580282 last 1788580252 [ 6711.861075] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6711.891366] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6711.918485] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6711.922514] Lustre: Skipped 1 previous similar message [ 6711.959630] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6720.482172] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6720.502766] LustreError: Skipped 8 previous similar messages [ 6733.280135] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6733.380050] Lustre: server umount lustre-OST0001 complete [ 6743.351889] Lustre: server umount lustre-OST0000 complete [ 6753.180993] Lustre: server umount lustre-MDT0000 complete [ 6757.496921] LustreError: 233012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788580330 with bad export cookie 8689989345863300681 [ 6757.500699] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6757.507380] LustreError: 233012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6757.986056] Lustre: server umount lustre-MDT0001 complete [ 6770.592871] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 6773.263379] Key type lgssc unregistered [ 6773.550440] LNet: 237901:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6773.556852] LNetError: 237901:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6773.566812] LNet: Removed LNI 192.168.203.122@tcp [ 6774.464273] Key type .llcrypt unregistered [ 6774.467074] Key type ._llcrypt unregistered [ 6799.215421] Key type ._llcrypt registered [ 6799.218406] Key type .llcrypt registered [ 6799.318210] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 6800.226868] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6800.324218] alg: No test for adler32 (adler32-zlib) [ 6801.364304] Lustre: Lustre: Build Version: 2.17.58_4_g27fe0b7 [ 6801.538444] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 6803.168468] Key type lgssc registered [ 6804.089814] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6813.979980] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6813.999615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6815.567311] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6820.078663] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6825.961178] LustreError: 239366:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6825.977795] LustreError: 239366:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6827.848406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6828.035985] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6828.080980] Lustre: 239354:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6828.090162] Lustre: Permanently deactivating lustre-MDT0001 [ 6828.103334] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6828.395449] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6832.664481] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6832.671997] LustreError: 239396:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 6832.687263] LustreError: 239396:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 6832.708852] LustreError: 239396:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 17, retries 1, failed: rc = -108 [ 6832.906886] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6839.816128] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6845.912862] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6852.436929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6852.907125] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6853.930378] LustreError: 241205:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6859.244790] LustreError: 241278:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6859.253448] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6860.297635] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 6865.383921] LustreError: 241208:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6865.410602] LustreError: 241208:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6866.558908] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6870.500463] LustreError: 241205:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6870.509315] LustreError: 241205:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6872.944895] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6873.118648] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6878.237593] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 6878.257658] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 6879.672848] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6889.630072] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6893.605103] Lustre: Permanently reactivating lustre-MDT0001 [ 6893.645069] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 6893.653245] Lustre: Skipped 2 previous similar messages [ 6897.173859] Lustre: lustre-MDT0001: Client cbbc2e5a-8311-46c1-a04e-b621251a8f77 (at 192.168.203.22@tcp) reconnecting [ 6905.029930] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6906.900749] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6914.724577] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6915.058710] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6921.813148] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6922.109718] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6925.356983] Lustre: Permanently deactivating lustre-MDT0001 [ 6925.389974] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6925.397250] Lustre: Skipped 2 previous similar messages [ 6934.695046] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6944.743470] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6944.758404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6944.773277] Lustre: Skipped 2 previous similar messages [ 6944.783272] Lustre: Skipped 3 previous similar messages [ 6949.253307] Lustre: server umount lustre-MDT0000 complete [ 6949.859867] LustreError: 239366:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6949.884397] LustreError: 239366:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6953.040313] LustreError: 239346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788580525 with bad export cookie 16502389649224384755 [ 6953.043632] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6953.065838] LustreError: 239346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6953.456996] Lustre: server umount lustre-MDT0001 complete [ 6959.608757] Lustre: server umount lustre-OST0000 complete [ 6971.423101] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:55:41 (1788580541) [ 6982.263357] Lustre: server umount lustre-OST0001 complete [ 6989.656239] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 6999.716511] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 7037.480227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7037.768550] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7037.777403] Lustre: Skipped 2 previous similar messages [ 7037.813281] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7037.933333] Lustre: lustre-MDT0000: new disk, initializing [ 7038.032214] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7038.045895] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7043.212127] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7052.750158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7052.922868] Lustre: 247639:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7052.989281] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7052.992659] Lustre: Skipped 1 previous similar message [ 7053.051804] Lustre: lustre-MDT0001: new disk, initializing [ 7053.160719] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7053.210115] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7053.224445] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7058.383466] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7069.744663] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7076.704354] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7084.763709] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7084.976526] Lustre: lustre-OST0000: new disk, initializing [ 7084.979461] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7084.983694] Lustre: 249609:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7085.064597] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7092.997606] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7093.880597] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7093.904948] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7094.006449] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7103.235788] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7105.181406] LustreError: 247647:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7114.122540] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7114.251173] Lustre: lustre-OST0001: new disk, initializing [ 7114.254948] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7114.260939] Lustre: 250728:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7115.200356] LustreError: 247647:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7115.255602] LustreError: 249057:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7121.496341] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7122.563431] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7122.579491] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7122.705306] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7125.328100] LustreError: 249057:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7132.726465] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7134.530633] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7138.279556] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7138.288853] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7138.300745] Lustre: Skipped 2 previous similar messages [ 7138.321337] Lustre: Skipped 1 previous similar message [ 7143.405505] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7143.420890] Lustre: Skipped 1 previous similar message [ 7148.522953] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7148.544775] Lustre: Skipped 1 previous similar message [ 7151.072189] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7151.285353] Lustre: server umount lustre-OST0001 complete [ 7153.640458] LustreError: 249592:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7153.656530] LustreError: 249592:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7169.007970] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7169.017384] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7169.025559] Lustre: Skipped 1 previous similar message [ 7169.042532] Lustre: Skipped 1 previous similar message [ 7179.234628] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7179.249085] Lustre: Skipped 3 previous similar messages [ 7183.328472] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7183.534033] Lustre: server umount lustre-OST0000 complete [ 7189.479090] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7193.669678] Lustre: server umount lustre-MDT0000 complete [ 7194.599943] LustreError: 247650:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7194.621302] LustreError: 247650:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 7197.676526] LustreError: 250727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788580770 with bad export cookie 16502389649224387352 [ 7197.685321] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7197.692993] LustreError: 250727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7204.406935] Lustre: server umount lustre-MDT0001 complete [ 7219.198445] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 7222.566921] Key type lgssc unregistered [ 7222.942739] LNet: 252764:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7222.954063] LNetError: 252764:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7222.969736] LNet: Removed LNI 192.168.203.122@tcp [ 7224.020514] Key type .llcrypt unregistered [ 7224.022793] Key type ._llcrypt unregistered [ 7267.304992] Key type ._llcrypt registered [ 7267.307698] Key type .llcrypt registered [ 7267.442073] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 7268.717127] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7268.759088] alg: No test for adler32 (adler32-zlib) [ 7269.808086] Lustre: Lustre: Build Version: 2.17.58_4_g27fe0b7 [ 7270.115206] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 7271.808183] Key type lgssc registered [ 7273.049646] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7281.530752] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7281.577266] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7282.895458] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7282.918647] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7283.155624] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7288.340125] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7299.840428] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7300.056084] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7300.110786] Lustre: 254972:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7300.345400] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7305.925377] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7313.596170] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7319.115300] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7326.026282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7326.194984] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7326.373200] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7333.196991] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7335.089535] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7341.091945] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7396.321436] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7396.350637] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7396.384964] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7396.852716] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7396.866512] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7396.885052] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7398.307180] Lustre: server umount lustre-OST0000 complete [ 7403.647485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7416.766556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7417.107823] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7422.323839] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7429.105264] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7436.531794] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7436.552563] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7436.557200] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7436.581335] Lustre: Skipped 1 previous similar message [ 7472.490192] Lustre: server umount lustre-OST0000 complete [ 7477.732376] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7477.760129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7482.854205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7482.872028] Lustre: Skipped 2 previous similar messages [ 7487.023220] Lustre: server umount lustre-MDT0000 complete [ 7489.056181] Lustre: 254293:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788581045/real 1788581045] req@000000002fbfe401 x1875462946101248/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788581061 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7489.085542] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7489.106846] Lustre: Skipped 1 previous similar message [ 7490.975675] LustreError: 254965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788581063 with bad export cookie 570645332812100875 [ 7490.981909] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7490.996883] LustreError: 254965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7491.536507] Lustre: server umount lustre-MDT0001 complete [ 7501.883244] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 7504.154330] Key type lgssc unregistered [ 7504.406037] LNet: 260408:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7504.416444] LNetError: 260408:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7504.434617] LNet: Removed LNI 192.168.203.122@tcp [ 7505.176912] Key type .llcrypt unregistered [ 7505.179938] Key type ._llcrypt unregistered [ 7528.454490] Key type ._llcrypt registered [ 7528.457151] Key type .llcrypt registered [ 7528.523324] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 7529.611270] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7529.653054] alg: No test for adler32 (adler32-zlib) [ 7530.743222] Lustre: Lustre: Build Version: 2.17.58_4_g27fe0b7 [ 7531.070653] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 7532.752202] Key type lgssc registered [ 7533.681819] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7541.483808] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7541.507619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7543.011114] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7547.863753] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7552.484406] LustreError: 261819:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7552.519348] LustreError: 261819:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7556.586609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7556.916631] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7561.290647] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7569.273968] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7576.381733] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7584.237559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7584.550379] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7587.648280] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7591.271131] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7599.226276] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7627.745104] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7627.749663] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7627.765206] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7627.778093] Lustre: Skipped 1 previous similar message [ 7627.793244] Lustre: Skipped 1 previous similar message [ 7629.985914] Lustre: server umount lustre-OST0000 complete [ 7633.895823] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7633.903294] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7633.926501] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7637.988633] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7637.990721] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7638.003916] Lustre: Skipped 1 previous similar message [ 7639.906305] Lustre: server umount lustre-MDT0000 complete [ 7643.106653] LustreError: 261819:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7643.121398] LustreError: 261819:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7643.123200] LustreError: 261800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788581216 with bad export cookie 9301161455940193468 [ 7643.129571] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7643.147564] LustreError: 261800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7643.455511] Lustre: server umount lustre-MDT0001 complete [ 7653.050611] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 7655.407757] Key type lgssc unregistered [ 7655.665763] LNet: 266125:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7655.670910] LNetError: 266125:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7655.681610] LNet: Removed LNI 192.168.203.122@tcp [ 7656.457936] Key type .llcrypt unregistered [ 7656.463353] Key type ._llcrypt unregistered [ 7673.040136] Key type ._llcrypt registered [ 7673.042319] Key type .llcrypt registered [ 7673.205103] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 7674.287388] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7674.445561] alg: No test for adler32 (adler32-zlib) [ 7675.550994] Lustre: Lustre: Build Version: 2.17.58_4_g27fe0b7 [ 7675.823104] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 7677.562993] Key type lgssc registered [ 7678.579214] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7686.456480] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7686.482155] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7688.007447] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7692.470327] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7697.384210] LustreError: 267341:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7697.400919] LustreError: 267341:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7700.457913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7700.903394] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7704.476239] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7711.427037] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7716.462553] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7722.268153] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7722.570797] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7728.523552] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7732.859566] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7735.875122] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7748.591543] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7748.594460] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7748.624792] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7748.632271] Lustre: Skipped 1 previous similar message [ 7753.427702] Lustre: server umount lustre-OST0000 complete [ 7758.819610] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7758.842885] Lustre: Skipped 1 previous similar message [ 7758.853353] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7758.862322] Lustre: Skipped 1 previous similar message [ 7763.780719] Lustre: server umount lustre-MDT0000 complete [ 7763.942407] LustreError: 267342:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7763.960508] LustreError: 267342:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7767.517218] LustreError: 267323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788581340 with bad export cookie 7161354502054539290 [ 7767.524920] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7767.530611] LustreError: 267323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7767.936853] Lustre: server umount lustre-MDT0001 complete [ 7780.742409] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 7783.844938] Key type lgssc unregistered [ 7784.206912] LNet: 271022:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7784.216418] LNetError: 271022:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7784.237555] LNet: Removed LNI 192.168.203.122@tcp [ 7785.540760] Key type .llcrypt unregistered [ 7785.546521] Key type ._llcrypt unregistered [ 7813.872376] Key type ._llcrypt registered [ 7813.874242] Key type .llcrypt registered [ 7813.988393] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 7815.705841] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7815.744942] alg: No test for adler32 (adler32-zlib) [ 7816.955700] Lustre: Lustre: Build Version: 2.17.58_4_g27fe0b7 [ 7817.259589] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 7818.984173] Key type lgssc registered [ 7820.445155] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7829.593906] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7829.620876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7831.281120] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7836.682222] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7841.763647] LustreError: 272438:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7841.774652] LustreError: 272438:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7846.003896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7846.499271] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7851.768627] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7860.397103] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7867.963359] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7875.181192] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7875.660876] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7876.732938] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7882.095491] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7890.492357] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7918.050538] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7918.056666] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7918.059486] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7918.062764] Lustre: Skipped 1 previous similar message [ 7922.358547] Lustre: server umount lustre-OST0000 complete [ 7928.298337] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7928.303019] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7928.310499] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7928.310508] Lustre: Skipped 1 previous similar message [ 7928.355907] Lustre: Skipped 1 previous similar message [ 7932.164714] Lustre: server umount lustre-MDT0000 complete [ 7933.414336] LustreError: 273634:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7933.455701] LustreError: 273634:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7935.824317] LustreError: 273176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788581508 with bad export cookie 3950908497959645104 [ 7935.828021] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7935.846169] LustreError: 273176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7936.189437] Lustre: server umount lustre-MDT0001 complete [ 7948.524771] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 7951.757941] Key type lgssc unregistered [ 7952.185314] LNet: 276744:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7952.191624] LNetError: 276744:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7952.221288] LNet: Removed LNI 192.168.203.122@tcp [ 7953.418884] Key type .llcrypt unregistered [ 7953.421793] Key type ._llcrypt unregistered [ 7973.555629] Key type ._llcrypt registered [ 7973.557255] Key type .llcrypt registered [ 7973.728906] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 7974.993198] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7975.031087] alg: No test for adler32 (adler32-zlib) [ 7976.121310] Lustre: Lustre: Build Version: 2.17.58_4_g27fe0b7 [ 7976.419774] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 7978.114086] Key type lgssc registered [ 7979.154230] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7986.397580] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7986.411338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7987.878931] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7992.540241] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7997.418152] LustreError: 277976:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7997.432437] LustreError: 277976:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7999.857006] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8000.196057] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8004.915972] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8012.747720] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8018.881616] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8025.795544] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8026.025808] LustreError: 279822:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8026.190721] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8032.154534] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8032.254503] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8039.848117] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8052.712573] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8052.718190] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8052.732675] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8052.753946] Lustre: Skipped 1 previous similar message [ 8057.835171] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8057.839168] Lustre: Skipped 1 previous similar message [ 8058.708590] Lustre: server umount lustre-OST0000 complete [ 8062.950981] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8062.971131] Lustre: Skipped 1 previous similar message [ 8062.980333] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8062.990760] Lustre: Skipped 1 previous similar message [ 8068.066680] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8068.078371] Lustre: Skipped 1 previous similar message [ 8068.662234] Lustre: server umount lustre-MDT0000 complete [ 8072.428855] LustreError: 277943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788581645 with bad export cookie 13791311254727440174 [ 8072.433241] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8072.439935] LustreError: 277943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8072.687103] Lustre: server umount lustre-MDT0001 complete [ 8084.927428] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 8087.559170] Key type lgssc unregistered [ 8087.802667] LNet: 281676:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8087.810303] LNetError: 281676:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8087.824263] LNet: Removed LNI 192.168.203.122@tcp [ 8088.542781] Key type .llcrypt unregistered [ 8088.544756] Key type ._llcrypt unregistered [ 8100.088511] Key type ._llcrypt registered [ 8100.092466] Key type .llcrypt registered [ 8100.228829] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8118.673639] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 8134.344226] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 8135.308426] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8135.428399] alg: No test for adler32 (adler32-zlib) [ 8136.457218] Lustre: Lustre: Build Version: 2.17.58_4_g27fe0b7 [ 8136.730379] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 8138.392163] Key type lgssc registered [ 8139.412909] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8175.270890] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8175.326134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8176.668119] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8176.730415] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8176.894043] Lustre: lustre-MDT0000: new disk, initializing [ 8176.990103] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8177.026849] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8181.425417] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8191.905439] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8192.068842] Lustre: 285229:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8192.140182] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8192.145403] Lustre: Skipped 1 previous similar message [ 8192.208121] Lustre: lustre-MDT0001: new disk, initializing [ 8192.284533] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8192.349602] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8192.356565] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8196.716020] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8206.687227] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8212.909508] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8221.187445] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8221.361650] Lustre: lustre-OST0000: new disk, initializing [ 8221.365773] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8221.371137] Lustre: 287197:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8221.462341] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8226.919848] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8226.923760] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8227.013730] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8228.019662] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8239.977669] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8244.441964] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8244.662220] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8249.921795] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8250.136847] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8252.897853] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8252.904287] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8252.915990] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8252.922531] Lustre: Skipped 1 previous similar message [ 8258.031768] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8258.038810] Lustre: Skipped 1 previous similar message [ 8258.858383] Lustre: server umount lustre-OST0000 complete [ 8263.137918] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8263.166382] Lustre: Skipped 1 previous similar message [ 8263.174343] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8268.257569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8268.270040] Lustre: Skipped 2 previous similar messages [ 8269.187577] Lustre: server umount lustre-MDT0000 complete [ 8273.850191] LustreError: 286023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788581846 with bad export cookie 12926780217289920190 [ 8273.856597] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8273.869095] LustreError: 286023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8274.174995] Lustre: server umount lustre-MDT0001 complete [ 8284.580951] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 00:17:34 (1788581854) [ 8292.255792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8292.822143] LustreError: 289610:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8292.851306] LustreError: 289610:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8292.955730] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8297.954864] LustreError: 289611:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8298.249837] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8303.077372] LustreError: 289610:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8308.196260] LustreError: 289611:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8308.477692] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8308.994216] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8313.725642] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8323.129225] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8330.033502] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8337.763787] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8338.145869] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8344.289575] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8352.180427] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8364.008659] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8364.011071] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8364.038801] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8364.051140] Lustre: Skipped 2 previous similar messages [ 8364.064083] Lustre: Skipped 1 previous similar message [ 8374.256052] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8374.274498] Lustre: Skipped 3 previous similar messages [ 8377.312521] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8377.631609] Lustre: server umount lustre-OST0000 complete [ 8384.485771] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8384.502782] Lustre: Skipped 1 previous similar message [ 8387.785371] Lustre: server umount lustre-MDT0000 complete [ 8389.605878] LustreError: 289605:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8389.626079] LustreError: 289605:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8392.159908] LustreError: 289591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788581965 with bad export cookie 12926780217289921072 [ 8392.176434] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8392.185100] LustreError: 289591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8392.630969] Lustre: server umount lustre-MDT0001 complete [ 8404.007528] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 8406.925471] Key type lgssc unregistered [ 8407.218418] LNet: 293147:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8407.231755] LNetError: 293147:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8407.258683] LNet: Removed LNI 192.168.203.122@tcp [ 8408.275038] Key type .llcrypt unregistered [ 8408.284366] Key type ._llcrypt unregistered [ 8435.942659] Key type ._llcrypt registered [ 8435.947446] Key type .llcrypt registered [ 8436.087156] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 8452.902124] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 8453.710620] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8453.819442] alg: No test for adler32 (adler32-zlib) [ 8455.046754] Lustre: Lustre: Build Version: 2.17.58_4_g27fe0b7 [ 8455.330653] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 8457.112190] Key type lgssc registered [ 8458.644358] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8503.645517] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8503.670542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8504.839169] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8504.879966] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8504.942944] Lustre: lustre-MDT0000: new disk, initializing [ 8505.013345] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8505.022255] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8508.673386] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8519.329600] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8519.470933] Lustre: 296973:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8519.582756] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8519.587921] Lustre: Skipped 1 previous similar message [ 8519.672354] Lustre: lustre-MDT0001: new disk, initializing [ 8519.781734] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8519.825835] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8519.846781] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8524.126974] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8534.671819] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8541.943724] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8550.794205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8551.207602] Lustre: lustre-OST2710: new disk, initializing [ 8551.213677] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8551.229400] Lustre: 298942:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8551.390595] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8557.894290] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8561.214541] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8561.224848] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8561.338780] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8566.545089] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8572.923067] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8573.041241] Lustre: lustre-OST03e8: new disk, initializing [ 8573.043897] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8573.049074] Lustre: 300005:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8573.117654] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8578.490440] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8580.250558] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8580.261176] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8580.395494] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8586.007400] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8595.046553] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8595.474026] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8601.693774] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8602.041280] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8607.344503] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8607.673433] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8612.943296] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8613.170779] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8619.653118] Lustre: server umount lustre-MDT0000 complete [ 8621.551127] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8621.570688] Lustre: Skipped 3 previous similar messages [ 8623.881440] LustreError: 300015:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788582196 with bad export cookie 12364958560641311464 [ 8623.891168] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8623.895805] LustreError: 300015:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8624.269675] Lustre: server umount lustre-MDT0001 complete [ 8638.927108] Lustre: server umount lustre-OST2710 complete [ 8653.194982] Lustre: server umount lustre-OST03e8 complete [ 8675.120469] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 8686.356733] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 8720.827445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8721.062722] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8721.093402] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8721.199705] Lustre: lustre-MDT0000: new disk, initializing [ 8721.277366] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8721.293584] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8726.060903] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8736.060857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8736.206582] Lustre: 305255:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8736.268851] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8736.274470] Lustre: Skipped 1 previous similar message [ 8736.384279] Lustre: lustre-MDT0001: new disk, initializing [ 8736.472251] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8736.513519] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8736.530591] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8740.229107] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8749.078875] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8756.134703] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8763.087877] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8763.403585] Lustre: lustre-OST0000: new disk, initializing [ 8763.411592] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8763.423967] Lustre: 307224:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8763.516852] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8769.404709] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8771.647779] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8771.663731] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8771.753328] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8779.404599] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8784.677836] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8784.858240] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8789.871240] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8790.112609] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8792.570217] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8792.576905] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8792.590515] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8792.615985] LustreError: Skipped 1 previous similar message [ 8797.673910] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8797.684017] Lustre: Skipped 2 previous similar messages [ 8798.571624] Lustre: server umount lustre-OST0000 complete [ 8802.786824] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8802.799181] Lustre: Skipped 1 previous similar message [ 8802.807427] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8802.814637] Lustre: Skipped 1 previous similar message [ 8807.906290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8807.910668] Lustre: Skipped 1 previous similar message [ 8808.894688] Lustre: server umount lustre-MDT0000 complete [ 8812.127519] LustreError: 305246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788582385 with bad export cookie 12364958560641313144 [ 8812.141076] LustreError: 305246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8812.143971] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8812.467485] Lustre: server umount lustre-MDT0001 complete [ 8820.765378] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 00:26:30 (1788582390) [ 8822.261338] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8824.118826] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 00:26:34 (1788582394) [ 8828.165983] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 8854.624654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8854.840919] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8854.854444] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8854.991753] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8859.493626] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8868.130970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8868.220751] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8868.244805] Lustre: 311078:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8873.010194] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8894.762261] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8895.015537] Lustre: Denying initial registration attempt from nid 192.168.203.122@tcp, specified as failover [ 8895.025552] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 8895.039109] LustreError: 312532:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -99 [ 8895.052605] LustreError: 312532:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 8895.057088] LustreError: 312532:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 8895.159258] Lustre: server umount lustre-OST0000 complete [ 8895.164580] LustreError: 312532:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 8903.655240] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8903.656746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8903.675939] Lustre: Skipped 2 previous similar messages [ 8907.925770] Lustre: server umount lustre-MDT0000 complete [ 8908.770670] LustreError: 312084:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8908.785102] LustreError: 312084:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 8911.891237] LustreError: 311072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788582484 with bad export cookie 12364958560641314019 [ 8911.893355] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8911.899445] LustreError: 311072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8912.157482] Lustre: server umount lustre-MDT0001 complete [ 8922.899481] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 8925.720339] Key type lgssc unregistered [ 8925.997886] LNet: 313652:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8926.009178] LNetError: 313652:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8926.026567] LNet: Removed LNI 192.168.203.122@tcp [ 8926.900506] Key type .llcrypt unregistered [ 8926.903189] Key type ._llcrypt unregistered [ 8938.354715] Key type ._llcrypt registered [ 8938.357686] Key type .llcrypt registered [ 8938.476626] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 8939.671602] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8939.684475] alg: No test for adler32 (adler32-zlib) [ 8940.776476] Lustre: Lustre: Build Version: 2.17.58_4_g27fe0b7 [ 8941.131307] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 8942.848232] Key type lgssc registered [ 8943.988738] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8994.051544] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 9003.526051] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9003.566910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9004.813083] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9004.834931] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9004.985455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9009.440150] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 9018.537305] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9018.688926] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9018.730850] Lustre: 316411:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9019.018683] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9023.936855] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 9032.436053] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9039.270504] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9058.996290] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9059.196819] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9059.394388] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9065.294620] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 9072.506396] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9078.245770] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9078.250912] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9078.254085] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9078.261063] Lustre: Skipped 1 previous similar message [ 9080.363795] Lustre: server umount lustre-OST0000 complete [ 9085.409136] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9085.416958] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9085.442789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9085.447542] Lustre: Skipped 1 previous similar message [ 9088.487975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9088.494816] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9093.602950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9093.618511] Lustre: Skipped 2 previous similar messages [ 9098.208308] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9098.610581] Lustre: server umount lustre-MDT0000 complete [ 9102.627166] LustreError: 318281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788582675 with bad export cookie 5424529931042250823 [ 9102.627563] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9102.634032] LustreError: 318281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9102.954790] Lustre: server umount lustre-MDT0001 complete [ 9114.588697] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 9117.708857] Key type lgssc unregistered [ 9118.125461] LNet: 319991:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9118.132602] LNetError: 319991:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9118.151537] LNet: Removed LNI 192.168.203.122@tcp [ 9119.026667] Key type .llcrypt unregistered [ 9119.029395] Key type ._llcrypt unregistered [ 9145.559487] Key type ._llcrypt registered [ 9145.562866] Key type .llcrypt registered [ 9145.737613] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 9147.650678] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9147.800430] alg: No test for adler32 (adler32-zlib) [ 9149.042656] Lustre: Lustre: Build Version: 2.17.58_4_g27fe0b7 [ 9149.268391] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 9150.920367] Key type lgssc registered [ 9152.790357] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9159.677451] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9159.702232] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9161.267748] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9165.128711] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 9171.441029] LustreError: 321404:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9171.452518] LustreError: 321404:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9172.819438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9173.204908] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9176.573639] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 9183.880075] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9189.008815] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9197.359946] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9197.685820] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9201.962845] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 9209.134995] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9219.554420] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9219.565359] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9219.595433] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9222.303487] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.22@tcp (stopping) [ 9222.630118] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9225.885955] Lustre: server umount lustre-MDT0000 complete [ 9227.392255] LustreError: 321399:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9230.528619] LustreError: 322139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788582803 with bad export cookie 6220613630753851577 [ 9230.532653] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9230.538915] LustreError: 322139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9232.518842] LustreError: 321400:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9232.543675] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.22@tcp (stopping) [ 9232.545072] LustreError: 321400:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9232.559538] Lustre: Skipped 3 previous similar messages [ 9232.868886] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9232.879481] Lustre: Skipped 1 previous similar message [ 9235.873333] LustreError: 321400:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9235.873729] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9235.890036] LustreError: 321400:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9235.909411] Lustre: Skipped 1 previous similar message [ 9237.081579] Lustre: server umount lustre-MDT0001 complete [ 9243.745197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9251.250577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9255.626161] LustreError: 323237:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9255.689405] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9255.734169] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9259.637477] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 9265.958826] LustreError: 323238:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9265.961984] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9265.971584] LustreError: 323238:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9267.044638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9267.361016] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9271.061412] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 9272.809681] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9278.525506] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9279.958568] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9286.268550] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9287.857492] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9293.289554] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9293.293895] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9293.294594] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9293.304341] Lustre: Skipped 1 previous similar message [ 9298.624469] Lustre: server umount lustre-OST0000 complete [ 9303.521515] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9303.526037] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9303.540367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9303.540374] Lustre: Skipped 3 previous similar messages [ 9308.988873] Lustre: server umount lustre-MDT0000 complete [ 9313.180783] LustreError: 321386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788582886 with bad export cookie 6220613630753858724 [ 9313.182703] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9313.196184] LustreError: 321386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9313.735791] Lustre: server umount lustre-MDT0001 complete [ 9324.709280] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 9327.770382] Key type lgssc unregistered [ 9328.013356] LNet: 327665:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9328.021800] LNetError: 327665:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9328.033754] LNet: Removed LNI 192.168.203.122@tcp [ 9328.776652] Key type .llcrypt unregistered [ 9328.783286] Key type ._llcrypt unregistered [ 9352.426222] Key type ._llcrypt registered [ 9352.429651] Key type .llcrypt registered [ 9352.588314] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 9353.726308] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9353.785651] alg: No test for adler32 (adler32-zlib) [ 9354.973912] Lustre: Lustre: Build Version: 2.17.58_4_g27fe0b7 [ 9355.245909] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 9356.921408] Key type lgssc registered [ 9357.929599] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9364.325357] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9364.348806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9365.710527] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9369.451575] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 9375.218163] LustreError: 329086:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9375.239523] LustreError: 329086:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9377.283970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9377.706732] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9381.593966] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 9388.718528] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9394.480847] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9398.241147] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9398.254397] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9398.261838] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9401.313135] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9401.321662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9401.350650] Lustre: Skipped 1 previous similar message [ 9402.404209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9402.414290] Lustre: Skipped 1 previous similar message [ 9403.942141] Lustre: server umount lustre-MDT0000 complete [ 9407.287694] LustreError: 329823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788582980 with bad export cookie 15488595273484460885 [ 9407.291952] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9407.296035] LustreError: 329823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9407.602883] Lustre: server umount lustre-MDT0001 complete [ 9420.397268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9420.621651] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9420.652802] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9420.803464] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9425.127447] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 9433.557750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9433.661064] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9433.691877] Lustre: 331670:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9433.895486] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9437.879341] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 9444.656413] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9449.235506] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9458.603140] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9458.819712] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9458.827370] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9458.835586] LustreError: 333556:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [ 9458.841318] LustreError: 333556:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 9458.847439] LustreError: 333556:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9458.903381] Lustre: server umount lustre-OST0000 complete [ 9458.905551] LustreError: 333556:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9467.241675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9467.413809] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9467.603697] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9473.925298] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 9475.664088] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9480.383652] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9487.937586] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9488.032607] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9488.101660] Lustre: lustre-OST0001: new disk, initializing [ 9488.105907] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9488.111611] Lustre: 335096:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9488.196125] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9494.286427] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 9498.191372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9498.199933] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9498.299843] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9501.365448] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9508.838276] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9508.841375] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9508.845582] LustreError: Skipped 1 previous similar message [ 9508.863025] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9508.933594] Lustre: Skipped 1 previous similar message [ 9510.245718] Lustre: server umount lustre-OST0001 complete [ 9513.971980] LustreError: 334021:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9513.988354] LustreError: 334021:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9516.513951] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9516.515617] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9516.521365] LustreError: Skipped 1 previous similar message [ 9516.526662] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9516.537302] Lustre: Skipped 2 previous similar messages [ 9516.552100] Lustre: Skipped 1 previous similar message [ 9519.088179] LustreError: 336100:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9519.099441] LustreError: 336100:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9524.209255] LustreError: 334021:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9524.221760] LustreError: 334021:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9528.288444] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9528.648388] Lustre: server umount lustre-OST0000 complete [ 9534.433386] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9534.455526] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9534.460594] Lustre: Skipped 5 previous similar messages [ 9538.640747] Lustre: server umount lustre-MDT0000 complete [ 9539.557846] LustreError: 331682:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9539.569261] LustreError: 331682:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9541.388442] LustreError: 331665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788583114 with bad export cookie 15488595273484461445 [ 9541.389820] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9541.408047] LustreError: 331665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9541.682582] Lustre: server umount lustre-MDT0001 complete [ 9551.526485] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 9554.229697] Key type lgssc unregistered [ 9554.494386] LNet: 337056:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9554.506923] LNetError: 337056:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9554.519264] LNet: Removed LNI 192.168.203.122@tcp [ 9555.277271] Key type .llcrypt unregistered [ 9555.280377] Key type ._llcrypt unregistered [ 9623.894385] Key type ._llcrypt registered [ 9623.896670] Key type .llcrypt registered [ 9624.028584] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 9643.429304] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 9645.407772] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9645.488090] alg: No test for adler32 (adler32-zlib) [ 9646.447606] Lustre: Lustre: Build Version: 2.17.58_4_g27fe0b7 [ 9646.680261] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 9648.480214] Key type lgssc registered [ 9649.720527] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9684.130319] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9684.147683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9685.401627] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9685.437937] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9685.512538] Lustre: lustre-MDT0000: new disk, initializing [ 9685.567836] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9685.582587] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9689.803719] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 9698.867159] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9698.978238] Lustre: 341989:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9699.055555] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9699.066244] Lustre: Skipped 1 previous similar message [ 9699.137657] Lustre: lustre-MDT0001: new disk, initializing [ 9699.209865] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9699.240499] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9699.253206] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9703.368576] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 9713.635751] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9719.483850] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9726.385140] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9726.637637] Lustre: lustre-OST0000: new disk, initializing [ 9726.642637] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9726.652043] Lustre: 343957:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9726.762648] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9733.132461] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 9735.254314] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9735.271441] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9735.412352] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9742.171356] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9746.136633] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9746.358179] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9750.336875] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9750.485896] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9756.130962] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9756.132395] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9756.142398] LustreError: Skipped 1 previous similar message [ 9756.148239] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9756.173208] Lustre: Skipped 1 previous similar message [ 9756.208648] Lustre: Skipped 1 previous similar message [ 9761.257320] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9761.275068] Lustre: Skipped 1 previous similar message [ 9766.368235] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9766.370176] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9766.402455] Lustre: Skipped 1 previous similar message [ 9766.726663] Lustre: server umount lustre-OST0000 complete [ 9770.978389] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9770.991676] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9771.008167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9776.620171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9776.623511] Lustre: Skipped 3 previous similar messages [ 9776.990940] Lustre: server umount lustre-MDT0000 complete [ 9780.493912] LustreError: 341981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788583353 with bad export cookie 10538117422420416032 [ 9780.500474] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9780.506971] LustreError: 341981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9780.803578] Lustre: server umount lustre-MDT0001 complete [ 9789.193330] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 00:42:38 (1788583358) [ 9795.873259] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 00:42:45 (1788583365) [ 9803.392292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9803.827657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9807.684889] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 9808.865387] LustreError: 346713:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9808.879876] LustreError: 346713:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9814.003719] LustreError: 346712:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9816.448461] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9816.729172] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9821.914884] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 9830.986740] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9836.758136] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9843.393547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9843.691163] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9849.775388] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 9856.998672] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9868.795973] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 9870.362719] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9871.785640] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9873.271182] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 9874.825994] Lustre: Failing over lustre-MDT0000 [ 9875.094223] LustreError: 346707:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9875.166494] Lustre: server umount lustre-MDT0000 complete [ 9876.449974] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9876.459404] Lustre: Skipped 3 previous similar messages [ 9880.192029] LustreError: 348250:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9880.216284] LustreError: 348250:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9885.311310] LustreError: 346708:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9885.336638] LustreError: 346708:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9892.841454] Lustre: 340209:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788583449/real 1788583449] req@0000000072002114 x1875465437518720/t0(0) o400->MGC192.168.203.122@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788583465 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9892.860875] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9893.456887] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9896.575207] LustreError: 346709:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9896.593644] LustreError: 346709:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 9903.363455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9903.411736] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9906.819457] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9907.635196] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 9908.749816] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9908.810905] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 9908.867213] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9917.455186] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9918.981700] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9922.490969] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 9924.406885] Lustre: Failing over lustre-MDT0000 [ 9924.584205] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9924.587496] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9924.622092] LustreError: 346713:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9924.635340] LustreError: 346713:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 9924.699966] Lustre: server umount lustre-MDT0000 complete [ 9929.189479] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9929.213893] Lustre: Skipped 1 previous similar message [ 9942.792039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9942.963207] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9943.246908] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9943.293901] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9943.725809] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9947.540608] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 9948.651119] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9948.657990] Lustre: Skipped 2 previous similar messages [ 9948.686207] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 9948.735681] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 9956.480423] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9958.204879] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9964.003877] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9964.005164] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9964.010092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9964.019645] Lustre: Skipped 1 previous similar message [ 9969.657351] Lustre: server umount lustre-MDT0000 complete [ 9972.849714] LustreError: 346692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788583545 with bad export cookie 10538117422420418741 [ 9972.857868] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9972.861939] LustreError: 346692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9973.133118] Lustre: server umount lustre-MDT0001 complete [ 9986.978685] Lustre: server umount lustre-OST0000 complete [10000.595047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10001.170496] LustreError: 353071:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10001.195390] LustreError: 353071:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [10004.726330] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [10012.173477] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10012.472719] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10012.486138] Lustre: Skipped 1 previous similar message [10016.103889] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [10022.789979] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10027.736587] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10033.795748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10039.884340] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [10046.444828] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10046.484580] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10052.584099] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10052.595851] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10052.615326] Lustre: Skipped 1 previous similar message [10052.626299] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10052.631707] Lustre: Skipped 5 previous similar messages [10054.061490] Lustre: server umount lustre-OST0000 complete [10068.010536] Lustre: server umount lustre-MDT0000 complete [10070.966443] LustreError: 353052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788583643 with bad export cookie 10538117422420419490 [10070.971076] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10070.975588] LustreError: 353052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10071.272622] Lustre: server umount lustre-MDT0001 complete [10081.428605] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [10084.306211] Key type lgssc unregistered [10084.667373] LNet: 356601:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10084.675436] LNetError: 356601:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10084.709239] LNet: Removed LNI 192.168.203.122@tcp [10085.683862] Key type .llcrypt unregistered [10085.687823] Key type ._llcrypt unregistered [10112.160691] Key type ._llcrypt registered [10112.163291] Key type .llcrypt registered [10112.314279] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [10113.267523] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10113.445946] alg: No test for adler32 (adler32-zlib) [10114.513608] Lustre: Lustre: Build Version: 2.17.58_4_g27fe0b7 [10114.765773] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [10116.456173] Key type lgssc registered [10117.519045] Lustre: Echo OBD driver; http://www.lustre.org/ [10124.582436] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10124.603052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10125.988736] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10130.150963] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [10135.523222] LustreError: 358113:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10135.540088] LustreError: 358113:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10138.347239] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10138.725967] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10142.737493] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [10149.590490] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10154.868365] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10160.131433] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10160.392680] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10165.701958] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [10170.634608] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10173.514967] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10184.676913] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10184.684304] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10184.695502] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10186.211521] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10186.222198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10186.226735] Lustre: Skipped 1 previous similar message [10186.237993] Lustre: Skipped 1 previous similar message [10186.469447] Lustre: server umount lustre-MDT0000 complete [10189.845948] LustreError: 358093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788583762 with bad export cookie 3917315259883257241 [10189.848616] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10189.853684] LustreError: 358093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10190.323203] Lustre: server umount lustre-MDT0001 complete [10204.145842] Lustre: server umount lustre-OST0000 complete [10215.208483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10215.641591] LustreError: 361823:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10215.706821] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10219.950792] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [10221.032468] LustreError: 361824:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10225.141779] LustreError: 361823:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10228.556333] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10228.958828] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10233.045838] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [10239.949580] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10244.931224] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10250.540087] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10250.867536] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10256.164656] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [10261.115386] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10263.508428] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10275.297848] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10275.312978] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10275.328376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10275.340350] Lustre: Skipped 1 previous similar message [10281.580816] Lustre: server umount lustre-MDT0000 complete [10281.958028] LustreError: 364365:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10281.977897] LustreError: 364365:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10285.156620] LustreError: 361805:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788583858 with bad export cookie 3917315259883258354 [10285.161189] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10285.182264] LustreError: 361805:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10285.515918] Lustre: server umount lustre-MDT0001 complete [10299.700770] Lustre: server umount lustre-OST0000 complete [10314.696354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10315.277672] LustreError: 365735:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10315.365365] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10320.028461] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [10329.131538] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10333.892649] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [10340.726727] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10346.726178] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10353.645622] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10353.895882] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10353.901210] Lustre: Skipped 1 previous similar message [10359.699777] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [10361.071103] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10367.458418] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10374.881466] Lustre: server umount lustre-OST0000 complete [10380.770416] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10380.775320] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10380.785937] Lustre: Skipped 2 previous similar messages [10380.790936] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10380.799599] Lustre: Skipped 3 previous similar messages [10386.913629] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10386.916774] Lustre: Skipped 2 previous similar messages [10389.016742] Lustre: server umount lustre-MDT0000 complete [10392.033801] Lustre: 357426:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788583948/real 1788583948] req@00000000f82a2dbc x1875465928266368/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788583964 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10392.036822] LustreError: 366463:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10392.059229] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10392.059238] Lustre: Skipped 1 previous similar message [10392.083461] LustreError: 366463:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10392.992466] Lustre: 357423:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788583949/real 1788583949] req@000000007e0fca6b x1875465928267136/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788583965 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10393.204809] LustreError: 366470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788583966 with bad export cookie 3917315259883259376 [10393.216334] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10393.217050] LustreError: 366470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10393.656200] Lustre: server umount lustre-MDT0001 complete [10405.697122] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [10409.540635] Key type lgssc unregistered [10409.914981] LNet: 369267:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10409.919775] LNetError: 369267:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10409.944220] LNet: Removed LNI 192.168.203.122@tcp [10411.181181] Key type .llcrypt unregistered [10411.185569] Key type ._llcrypt unregistered [10440.801184] Key type ._llcrypt registered [10440.803913] Key type .llcrypt registered [10440.918977] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [10441.951477] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10442.000279] alg: No test for adler32 (adler32-zlib) [10443.055927] Lustre: Lustre: Build Version: 2.17.58_4_g27fe0b7 [10443.283586] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [10444.896188] Key type lgssc registered [10446.150476] Lustre: Echo OBD driver; http://www.lustre.org/ [10454.082379] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10454.107333] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10454.118097] LustreError: 370704:0:(osd_handler.c:8546:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10454.148557] LustreError: 370704:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10454.154551] LustreError: 370704:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10454.168551] LustreError: 370704:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10454.177145] LustreError: 370704:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10461.195612] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10461.203221] LustreError: 371101:0:(osd_handler.c:8546:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10461.234085] LustreError: 371101:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10461.241664] LustreError: 371101:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10461.253956] LustreError: 371101:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10461.263157] LustreError: 371101:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10480.839633] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [10483.136562] Key type lgssc unregistered [10483.468523] LNet: 372102:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10483.478290] LNetError: 372102:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10483.494251] LNet: Removed LNI 192.168.203.122@tcp [10484.319442] Key type .llcrypt unregistered [10484.321853] Key type ._llcrypt unregistered [10500.633293] Key type ._llcrypt registered [10500.634885] Key type .llcrypt registered [10500.705483] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [10515.690655] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [10516.969744] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10517.004397] alg: No test for adler32 (adler32-zlib) [10518.020865] Lustre: Lustre: Build Version: 2.17.58_4_g27fe0b7 [10518.260849] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [10519.904198] Key type lgssc registered [10520.942622] Lustre: Echo OBD driver; http://www.lustre.org/ [10558.362034] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10558.409966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10559.696210] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10559.735043] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10559.875967] Lustre: lustre-MDT0000: new disk, initializing [10559.996283] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10560.019194] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10564.760191] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [10575.200266] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10575.311297] Lustre: 375256:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10575.360885] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10575.364385] Lustre: Skipped 1 previous similar message [10575.431646] Lustre: lustre-MDT0001: new disk, initializing [10575.523856] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10575.566159] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10575.580139] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10579.590256] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [10588.086572] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10593.566787] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10600.879547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10601.176935] Lustre: lustre-OST0000: new disk, initializing [10601.181845] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10601.196785] Lustre: 377225:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10601.273632] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10606.636478] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10606.654115] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10606.778545] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10607.480144] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [10617.034370] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10622.059180] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10622.314279] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10627.236704] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10627.499849] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10632.680208] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10632.686038] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10632.692654] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10632.708069] LustreError: Skipped 1 previous similar message [10632.752202] Lustre: Skipped 1 previous similar message [10635.873302] Lustre: server umount lustre-OST0000 complete [10642.406367] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10642.412278] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10642.432274] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10642.437390] Lustre: Skipped 1 previous similar message [10645.811853] Lustre: server umount lustre-MDT0000 complete [10649.894553] LustreError: 376051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788584222 with bad export cookie 772648137687879934 [10649.914534] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10649.915424] LustreError: 376051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10650.340962] Lustre: server umount lustre-MDT0001 complete [10659.842299] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [10661.812284] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 00:57:11 (1788584231) [10668.250972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10668.779176] LustreError: 379684:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10668.794578] LustreError: 379684:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10668.851722] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10673.742329] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [10674.145709] LustreError: 379685:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10678.248106] LustreError: 379684:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10682.161919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10682.534578] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10687.732488] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [10697.049302] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10703.811346] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10710.990072] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10711.435372] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10717.712614] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [10726.233313] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10733.707049] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10733.888120] Lustre: lustre-OST0001: new disk, initializing [10733.894948] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10733.903471] Lustre: 382521:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10733.993502] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10740.040522] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [10740.331855] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10740.338626] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10740.423092] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10749.166267] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10756.069547] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10756.070177] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10756.081728] LustreError: Skipped 1 previous similar message [10756.083845] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10756.083853] Lustre: Skipped 1 previous similar message [10756.113598] Lustre: Skipped 2 previous similar messages [10759.260472] Lustre: server umount lustre-OST0001 complete [10761.191119] LustreError: 381506:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10761.220525] LustreError: 381506:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10766.311625] LustreError: 381505:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10766.337836] LustreError: 381505:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10776.560502] LustreError: 382903:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10776.562140] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10776.570164] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10776.570172] Lustre: Skipped 2 previous similar messages [10776.583964] LustreError: 382903:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10776.594746] Lustre: Skipped 1 previous similar message [10781.666262] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10781.671594] Lustre: Skipped 2 previous similar messages [10782.653368] Lustre: server umount lustre-OST0000 complete [10786.789045] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10786.806971] Lustre: Skipped 1 previous similar message [10791.905232] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10791.914109] Lustre: Skipped 3 previous similar messages [10792.743874] Lustre: server umount lustre-MDT0000 complete [10796.170900] LustreError: 379665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788584369 with bad export cookie 772648137687880816 [10796.171162] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10796.179649] LustreError: 379665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10802.578386] Lustre: server umount lustre-MDT0001 complete [10813.817880] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [10816.821707] Key type lgssc unregistered [10817.156687] LNet: 384510:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10817.165916] LNetError: 384510:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10817.184350] LNet: Removed LNI 192.168.203.122@tcp [10818.141521] Key type .llcrypt unregistered [10818.145478] Key type ._llcrypt unregistered [10866.706138] Key type ._llcrypt registered [10866.708276] Key type .llcrypt registered [10866.839376] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [10867.873473] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10867.895144] alg: No test for adler32 (adler32-zlib) [10868.987087] Lustre: Lustre: Build Version: 2.17.58_4_g27fe0b7 [10869.314932] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [10871.024170] Key type lgssc registered [10872.092970] Lustre: Echo OBD driver; http://www.lustre.org/ [10880.652371] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10880.672557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10881.926718] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10881.968940] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10882.161132] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10887.400371] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [10897.913804] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10898.000709] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10898.040717] Lustre: 387310:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10898.346893] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10903.314791] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [10910.417123] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10916.408436] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10918.885923] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10918.894246] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10918.909551] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10922.465575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10922.471659] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10924.628774] Lustre: server umount lustre-MDT0000 complete [10927.640550] LustreError: 387300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788584500 with bad export cookie 12346766153363098887 [10927.643182] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10927.650934] LustreError: 387300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10927.891604] Lustre: server umount lustre-MDT0001 complete [10937.011867] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 01:01:46 (1788584506) [10943.116636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10943.557865] LustreError: 389835:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10943.574536] LustreError: 389835:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10943.614789] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10947.954251] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [10949.091990] LustreError: 389836:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10954.216585] LustreError: 389835:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10956.135309] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10956.466714] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10960.269496] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [10966.975710] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10972.960305] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10979.137410] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10979.291953] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10979.480827] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10985.324373] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [10992.353848] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11000.861046] Lustre: Permanently deactivating lustre-OST0000 [11000.876362] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11006.495455] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11008.620800] LustreError: 392771:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11008.635678] LustreError: 392771:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11008.644231] LustreError: 392771:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11012.370472] LustreError: 392819:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11012.380973] LustreError: 392819:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11020.781311] Lustre: server umount lustre-OST0000 complete [11023.737936] LustreError: 393070:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11023.744239] LustreError: 393070:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11026.404977] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11026.407693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11026.427322] Lustre: Skipped 1 previous similar message [11026.444357] Lustre: Skipped 2 previous similar messages [11031.522561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11031.527404] Lustre: Skipped 1 previous similar message [11035.701136] Lustre: server umount lustre-MDT0000 complete [11036.642980] LustreError: 390560:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11036.679818] LustreError: 390560:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11039.489796] LustreError: 392524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788584612 with bad export cookie 12346766153363099468 [11039.492027] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11039.498876] LustreError: 392524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11050.178585] Lustre: server umount lustre-MDT0001 complete [11058.950966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11063.929470] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [11066.676940] LustreError: 394229:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11073.113914] Lustre: 394568:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.122@tcp [11075.374873] Lustre: server umount MGS complete [11086.252341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11086.742796] LustreError: 395292:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11086.768656] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11086.776532] Lustre: Skipped 1 previous similar message [11086.836224] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11090.870256] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [11097.069473] LustreError: 395293:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11097.085486] LustreError: 395293:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11098.874136] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11099.240059] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11104.100422] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [11112.144294] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11118.561758] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11120.491856] Lustre: Permanently reactivating lustre-OST0000 [11120.508270] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11120.516587] Lustre: Skipped 2 previous similar messages [11148.768891] LustreError: 397081:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11156.998981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11157.402771] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11157.408973] Lustre: Skipped 1 previous similar message [11163.871968] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [11172.407876] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11178.978420] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11178.978582] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11178.997039] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11178.999773] LustreError: Skipped 1 previous similar message [11179.010483] Lustre: Skipped 1 previous similar message [11180.053707] Lustre: server umount lustre-OST0000 complete [11184.100884] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11184.117049] Lustre: Skipped 2 previous similar messages [11189.217873] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11189.226463] Lustre: Skipped 3 previous similar messages [11190.100685] Lustre: server umount lustre-MDT0000 complete [11193.839045] LustreError: 396865:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788584766 with bad export cookie 12346766153363100511 [11193.840317] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11193.853591] LustreError: 396865:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11194.204213] Lustre: server umount lustre-MDT0001 complete [11205.180924] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [11208.036596] Key type lgssc unregistered [11208.364994] LNet: 399706:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11208.370252] LNetError: 399706:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11208.382756] LNet: Removed LNI 192.168.203.122@tcp [11209.357474] Key type .llcrypt unregistered [11209.360995] Key type ._llcrypt unregistered [11227.337729] Key type ._llcrypt registered [11227.341971] Key type .llcrypt registered [11227.463607] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [11246.650605] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [11247.750207] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11247.883734] alg: No test for adler32 (adler32-zlib) [11248.992039] Lustre: Lustre: Build Version: 2.17.58_4_g27fe0b7 [11249.290526] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [11250.984239] Key type lgssc registered [11252.139727] Lustre: Echo OBD driver; http://www.lustre.org/ [11293.045068] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11293.097900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11294.596826] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11294.665568] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11294.815447] Lustre: lustre-MDT0000: new disk, initializing [11294.967969] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11295.028592] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11300.387239] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [11312.299721] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11312.479043] Lustre: 402861:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11312.545280] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11312.550179] Lustre: Skipped 1 previous similar message [11312.617666] Lustre: lustre-MDT0001: new disk, initializing [11312.726667] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11312.766348] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11312.787466] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11317.538422] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [11328.352807] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11335.976127] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11344.523530] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11345.072385] Lustre: lustre-OST0000: new disk, initializing [11345.087899] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11345.100598] Lustre: 404829:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11345.238181] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11352.353929] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [11353.187128] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11353.208426] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11353.363703] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11363.156611] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11368.651887] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11368.894746] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11374.305397] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11374.655698] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11379.170487] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11379.185307] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11379.204276] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11379.215372] Lustre: Skipped 1 previous similar message [11383.003132] Lustre: server umount lustre-OST0000 complete [11389.411678] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11389.419178] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11389.421610] LustreError: Skipped 1 previous similar message [11389.425303] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11389.443160] Lustre: Skipped 2 previous similar messages [11389.468288] Lustre: Skipped 1 previous similar message [11393.559318] Lustre: server umount lustre-MDT0000 complete [11397.729189] LustreError: 402852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788584970 with bad export cookie 1295785739779585251 [11397.741871] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11397.744664] LustreError: 402852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11398.407961] Lustre: server umount lustre-MDT0001 complete [11408.085629] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11410.279378] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 01:09:39 (1788584979) [11418.369549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11419.258634] LustreError: 407290:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11419.282775] LustreError: 407290:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11419.395467] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11424.748030] LustreError: 407291:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11425.272811] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [11429.861790] LustreError: 407290:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11434.977846] LustreError: 407291:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11436.356689] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11436.822202] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11441.939803] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [11450.951475] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11457.531294] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11465.903579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11466.249445] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11473.394066] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [11482.090985] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11484.753370] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11491.846075] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11500.003925] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11500.006991] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11500.010430] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11500.059774] Lustre: Skipped 1 previous similar message [11503.904601] Lustre: server umount lustre-OST0000 complete [11508.706292] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11508.711553] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11508.724205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11508.740186] Lustre: Skipped 1 previous similar message [11514.392951] Lustre: server umount lustre-MDT0000 complete [11515.366436] LustreError: 407290:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11515.394891] LustreError: 407290:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11518.670939] LustreError: 407271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788585091 with bad export cookie 1295785739779586133 [11518.674502] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11518.690219] LustreError: 407271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11519.236507] Lustre: server umount lustre-MDT0001 complete [11532.407656] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [11535.203378] Key type lgssc unregistered [11535.584966] LNet: 411019:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11535.597629] LNetError: 411019:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11535.627150] LNet: Removed LNI 192.168.203.122@tcp [11536.923966] Key type .llcrypt unregistered [11536.925828] Key type ._llcrypt unregistered [11563.989520] Key type ._llcrypt registered [11563.991427] Key type .llcrypt registered [11564.091610] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [11564.390534] Key type .llcrypt unregistered [11564.397643] Key type ._llcrypt unregistered [11584.047283] Key type ._llcrypt registered [11584.048961] Key type .llcrypt registered [11584.216470] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [11585.825102] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11586.107542] alg: No test for adler32 (adler32-zlib) [11587.451442] Lustre: Lustre: Build Version: 2.17.58_4_g27fe0b7 [11587.898789] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [11589.641324] Key type lgssc registered [11591.198986] Lustre: Echo OBD driver; http://www.lustre.org/ [11600.427613] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11600.501257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11602.164757] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11606.434940] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [11612.656440] LustreError: 413227:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11612.673858] LustreError: 413227:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11615.632635] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11616.046885] LustreError: 413226:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11616.079437] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11616.090471] LustreError: 413226:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11621.351464] LustreError: 413227:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11621.386597] LustreError: 413227:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11625.331963] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [11626.473352] LustreError: 413221:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11626.514845] LustreError: 413221:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11627.528053] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11632.611408] LustreError: 413226:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11632.643581] LustreError: 413226:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11636.489848] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11642.851586] LustreError: 413222:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11642.878540] LustreError: 413222:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11643.640571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11643.845716] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11648.818399] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [11658.136533] Lustre: server umount lustre-OST0000 complete [11664.354397] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11664.362434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11664.366723] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11664.380054] Lustre: Skipped 1 previous similar message [11664.407772] Lustre: Skipped 1 previous similar message [11669.478122] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11669.500453] Lustre: Skipped 1 previous similar message [11671.632513] Lustre: server umount lustre-MDT0000 complete [11674.599482] LustreError: 413226:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11674.617859] LustreError: 413226:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11675.111031] Lustre: 412535:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788585232/real 1788585232] req@00000000c66f5c97 x1875467472760832/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788585248 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11675.149659] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11675.619173] Lustre: 412537:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788585232/real 1788585232] req@00000000700d61fe x1875467472760192/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788585248 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11675.665219] LustreError: 413205:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788585248 with bad export cookie 16053557629048930318 [11675.673607] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11676.058310] Lustre: server umount lustre-MDT0001 complete [11687.731940] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [11690.353445] Key type lgssc unregistered [11690.585809] LNet: 416374:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11690.593549] LNetError: 416374:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11690.607388] LNet: Removed LNI 192.168.203.122@tcp [11691.280153] Key type .llcrypt unregistered [11691.284170] Key type ._llcrypt unregistered [11722.078611] Key type ._llcrypt registered [11722.080875] Key type .llcrypt registered [11722.203314] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [11723.261590] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11723.375771] alg: No test for adler32 (adler32-zlib) [11724.391076] Lustre: Lustre: Build Version: 2.17.58_4_g27fe0b7 [11724.639823] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [11726.296200] Key type lgssc registered [11727.439276] Lustre: Echo OBD driver; http://www.lustre.org/ [11736.490128] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11736.536051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11752.096820] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [11757.216348] LustreError: 417767:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.122@tcp: failed processing log, type 4: rc = -110 [11783.904201] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11791.037857] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [11799.861785] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11806.622236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11807.142320] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11807.212314] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11812.323278] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [11812.326196] LustreError: 417793:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11812.385885] LustreError: 417793:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11817.449787] LustreError: 417792:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11818.470192] LustreError: 418831:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11818.486317] LustreError: 418831:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11820.420544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11820.939340] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11826.452557] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [11837.504225] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11846.156760] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11853.289084] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11853.306563] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11853.323433] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11855.088427] Lustre: server umount lustre-OST0000 complete [11860.963847] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [11860.971203] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11860.985715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11861.263977] Lustre: server umount lustre-MDT0000 complete [11861.997948] LustreError: 418809:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11862.010777] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11864.732997] LustreError: 417775:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788585437 with bad export cookie 16987865838457506184 [11864.755909] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11864.758586] LustreError: 417775:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11874.588242] Lustre: server umount lustre-MDT0001 complete [11893.546515] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [11898.444972] Key type lgssc unregistered [11898.878723] LNet: 421359:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11898.883841] LNetError: 421359:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11898.905260] LNet: Removed LNI 192.168.203.122@tcp [11900.089197] Key type .llcrypt unregistered [11900.092707] Key type ._llcrypt unregistered [11926.791612] Key type ._llcrypt registered [11926.793613] Key type .llcrypt registered [11926.886878] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [11928.061986] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11928.128433] alg: No test for adler32 (adler32-zlib) [11929.179641] Lustre: Lustre: Build Version: 2.17.58_4_g27fe0b7 [11929.485666] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [11931.200181] Key type lgssc registered [11932.252595] Lustre: Echo OBD driver; http://www.lustre.org/ [11940.042940] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11940.072509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11941.442850] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11945.994905] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [11951.585800] LustreError: 422773:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11951.605112] LustreError: 422773:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11955.027331] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11955.345474] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11959.778491] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [11967.013552] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11973.218869] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11980.781135] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11981.116504] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11986.222590] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11986.960386] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [11995.686531] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12003.518612] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.22@tcp (stopping) [12003.730342] Lustre: server umount lustre-MDT0000 complete [12005.860764] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12005.865857] LustreError: 424612:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12005.894475] Lustre: Skipped 1 previous similar message [12005.968962] LustreError: 424612:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12007.412259] LustreError: 422755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.22@tcp arrived at 1788585580 with bad export cookie 2875528503203152980 [12007.426392] LustreError: 422755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12009.363029] LustreError: 422754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788585582 with bad export cookie 2875528503203152091 [12009.368579] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12009.373469] LustreError: 422754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12010.989412] LustreError: 422773:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12011.000797] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12011.013641] LustreError: 422773:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12011.041743] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12011.055881] Lustre: Skipped 1 previous similar message [12011.060408] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12015.013364] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12015.014807] LustreError: 423501:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12015.063275] LustreError: 423501:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12016.002053] Lustre: server umount lustre-OST0000 complete [12019.107561] LustreError: 422774:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12019.115262] LustreError: 422774:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12023.925365] Lustre: server umount lustre-MDT0001 complete [12037.697493] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [12040.655652] Key type lgssc unregistered [12040.999758] LNet: 426457:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12041.030808] LNetError: 426457:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12041.074748] LNet: Removed LNI 192.168.203.122@tcp [12042.132685] Key type .llcrypt unregistered [12042.134138] Key type ._llcrypt unregistered [12070.350053] Key type ._llcrypt registered [12070.357431] Key type .llcrypt registered [12070.486062] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [12071.200852] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12071.215465] alg: No test for adler32 (adler32-zlib) [12072.160907] Lustre: Lustre: Build Version: 2.17.58_4_g27fe0b7 [12072.452661] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [12074.136275] Key type lgssc registered [12075.150967] Lustre: Echo OBD driver; http://www.lustre.org/ [12082.649441] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12082.659235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12084.136665] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12088.488383] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [12094.439217] LustreError: 427870:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12094.461723] LustreError: 427870:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12097.838394] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12098.468846] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12104.779451] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [12112.452957] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12118.358472] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12127.271184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12127.529118] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12129.611326] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12134.821595] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [12143.061534] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12149.730867] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12149.735707] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12149.751835] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12150.250214] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12150.275379] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12150.280719] Lustre: Skipped 2 previous similar messages [12153.451874] Lustre: server umount lustre-MDT0001 complete [12155.009436] LustreError: 427867:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12160.485697] LustreError: 427865:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12160.518027] LustreError: 427865:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12165.603923] LustreError: 427871:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12165.604724] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12165.655488] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12165.676595] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12167.759630] Lustre: server umount lustre-OST0000 complete [12170.728212] LustreError: 428599:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12173.351613] Lustre: server umount lustre-MDT0000 complete [12187.771922] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [12191.080520] Key type lgssc unregistered [12191.431154] LNet: 431554:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12191.455374] LNetError: 431554:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12191.474882] LNet: Removed LNI 192.168.203.122@tcp [12192.636583] Key type .llcrypt unregistered [12192.638826] Key type ._llcrypt unregistered [12219.348427] Key type ._llcrypt registered [12219.351315] Key type .llcrypt registered [12219.434324] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [12232.789415] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [12233.606726] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12233.634197] alg: No test for adler32 (adler32-zlib) [12234.868661] Lustre: Lustre: Build Version: 2.17.58_4_g27fe0b7 [12235.226693] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [12236.944282] Key type lgssc registered [12238.119786] Lustre: Echo OBD driver; http://www.lustre.org/ [12275.273907] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [12289.117748] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12289.149043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12290.364298] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12290.394890] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12290.491412] Lustre: lustre-MDT0000: new disk, initializing [12290.584233] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12290.610565] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12294.694649] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [12310.930073] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12311.252513] Lustre: lustre-OST0000: new disk, initializing [12311.258809] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12311.264346] Lustre: Skipped 1 previous similar message [12311.269181] Lustre: 436596:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12311.344801] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12317.507459] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [12317.831645] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12317.846162] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12317.888221] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12327.381084] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12344.053239] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12344.212053] Lustre: 435580:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12344.293105] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12344.370889] Lustre: lustre-MDT0001: new disk, initializing [12344.499339] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12344.550447] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12344.568261] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12349.364849] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [12359.318949] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12359.529243] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12364.491858] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12364.730510] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12369.890155] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12369.900161] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12369.902594] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12369.932036] Lustre: Skipped 1 previous similar message [12369.947590] Lustre: Skipped 1 previous similar message [12371.107820] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.22@tcp (stopping) [12375.011469] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12375.023933] Lustre: Skipped 1 previous similar message [12376.239553] Lustre: server umount lustre-MDT0001 complete [12381.314132] LustreError: 435587:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12381.336373] LustreError: 435587:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12385.253273] LustreError: 435591:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12385.288936] LustreError: 435591:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12390.375425] LustreError: 435587:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12390.400965] LustreError: 435587:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12395.496714] LustreError: 435592:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12395.500853] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12395.537018] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12400.625595] LustreError: 435591:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12400.629224] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12401.684657] Lustre: server umount lustre-OST0000 complete [12412.794943] Lustre: server umount lustre-MDT0000 complete [12429.290349] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [12432.286872] Key type lgssc unregistered [12432.650868] LNet: 440349:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12432.685655] LNetError: 440349:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12432.727099] LNet: Removed LNI 192.168.203.122@tcp [12434.179239] Key type .llcrypt unregistered [12434.182925] Key type ._llcrypt unregistered [12479.231630] Key type ._llcrypt registered [12479.235250] Key type .llcrypt registered [12479.360639] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [12479.659578] Key type .llcrypt unregistered [12479.662782] Key type ._llcrypt unregistered [12498.293124] Key type ._llcrypt registered [12498.300831] Key type .llcrypt registered [12498.417626] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [12499.620728] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12499.636090] alg: No test for adler32 (adler32-zlib) [12500.596283] Lustre: Lustre: Build Version: 2.17.58_4_g27fe0b7 [12500.863580] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [12502.592215] Key type lgssc registered [12503.580871] Lustre: Echo OBD driver; http://www.lustre.org/ [12533.534474] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12533.550430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12534.779967] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12534.823349] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12534.912595] Lustre: lustre-MDT0000: new disk, initializing [12535.025451] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12535.045128] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12540.453291] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [12551.384769] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12551.561160] Lustre: 443728:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12551.667601] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12551.674891] Lustre: Skipped 1 previous similar message [12551.763984] Lustre: lustre-MDT0001: new disk, initializing [12551.901489] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12551.945853] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12551.963715] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12556.260563] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [12566.391718] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12573.370668] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12581.247102] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12581.550357] Lustre: lustre-OST0000: new disk, initializing [12581.561674] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12581.570923] Lustre: 445694:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12581.675889] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12587.123782] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12587.134973] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12587.261174] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12589.159846] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [12600.448492] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12613.095182] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12613.115151] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12613.128323] Lustre: Skipped 1 previous similar message [12613.152164] Lustre: Skipped 2 previous similar messages [12615.071934] Lustre: server umount lustre-MDT0000 complete [12618.216430] LustreError: 444841:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12618.257715] LustreError: 444841:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12619.398687] LustreError: 445693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788586192 with bad export cookie 9829952652014916835 [12619.408483] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12619.419204] LustreError: 445693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12619.842928] Lustre: server umount lustre-MDT0001 complete [12633.372659] Lustre: server umount lustre-OST0000 complete [12645.499551] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 01:30:15 (1788586215) [12655.323289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12655.892375] LustreError: 447725:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12655.983174] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12661.218658] LustreError: 447726:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12661.594843] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [12665.332660] LustreError: 447725:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12670.434790] LustreError: 447726:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12673.068941] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12673.511167] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12678.502420] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [12687.188717] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12693.874691] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12715.720490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12715.953540] Lustre: Found index 0 for lustre-OST0000, updating log [12716.150793] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12723.320756] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [12726.401953] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [12731.773866] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12739.042995] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12739.052997] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12739.078761] Lustre: Skipped 1 previous similar message [12739.086170] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12739.814371] Lustre: server umount lustre-OST0000 complete [12757.092890] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12757.102126] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12757.108036] Lustre: Skipped 2 previous similar messages [12757.122594] Lustre: Skipped 2 previous similar messages [12762.459695] Lustre: server umount lustre-MDT0000 complete [12762.597026] LustreError: 447726:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12762.619357] LustreError: 447726:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12766.817191] LustreError: 447707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788586339 with bad export cookie 9829952652014918235 [12766.831265] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12766.852820] LustreError: 447707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12767.245169] Lustre: server umount lustre-MDT0001 complete [12776.424783] LNet: 1 local NIs in recovery (showing 1): 192.168.203.122@tcp [12779.649479] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 01:32:28 (1788586348) [13397.503865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13397.933414] LustreError: 454135:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13398.025954] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13402.335553] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [13403.111916] LustreError: 454136:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13403.134400] LustreError: 454136:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13407.203226] LustreError: 454135:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13410.074803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13410.464479] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13414.721822] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [13422.382671] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13427.207130] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13446.263512] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13446.527325] Lustre: Found index 0 for lustre-OST0000, updating log [13446.812821] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13453.321288] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [13454.439508] LNet: 1 local NIs in recovery (showing 1): 192.168.203.122@tcp [13460.478363] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13462.341196] Lustre: server umount lustre-OST0000 complete [13466.596386] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13466.603128] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13466.618042] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13467.682272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13467.690512] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13468.706802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13468.721352] Lustre: Skipped 2 previous similar messages [13470.754984] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13470.762917] Lustre: Skipped 3 previous similar messages [13472.060094] Lustre: server umount lustre-MDT0000 complete [13472.804102] LustreError: 454135:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13472.829099] LustreError: 454135:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13475.772513] LustreError: 455985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788587048 with bad export cookie 9829952652014919313 [13475.776367] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13475.785343] LustreError: 455985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13476.040578] Lustre: server umount lustre-MDT0001 complete [13506.370496] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [13516.120378] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [13518.947849] LNet: 1 local NIs in recovery (showing 1): 192.168.203.122@tcp [13552.898733] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 01:45:22 (1788587122) [13570.354742] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [13572.906889] Key type lgssc unregistered [13573.228475] LNet: 460742:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13573.234093] LNetError: 460742:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13698.216815] LNet: Removed LNI 192.168.203.122@tcp [13698.971744] Key type .llcrypt unregistered [13698.977433] Key type ._llcrypt unregistered [13715.310089] Key type ._llcrypt registered [13715.311873] Key type .llcrypt registered [13715.421496] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [13716.605642] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13716.786152] alg: No test for adler32 (adler32-zlib) [13717.877288] Lustre: Lustre: Build Version: 2.17.58_4_g27fe0b7 [13718.206923] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [13719.944921] Key type lgssc registered [13720.870344] Lustre: Echo OBD driver; http://www.lustre.org/ [13725.109723] LNet: Added LNI 192.168.203.122@tcp100 [8/256/0/180] [13726.174196] LNet: Added LNI 192.168.203.122@tcp101 [8/256/0/180] [13727.017527] LNet: Added LNI 192.168.203.122@tcp102 [8/256/0/180] [13727.809993] LNet: Added LNI 192.168.203.122@tcp103 [8/256/0/180] [13728.709499] LNet: Added LNI 192.168.203.122@tcp104 [8/256/0/180] [13729.532794] LNet: Added LNI 192.168.203.122@tcp105 [8/256/0/180] [13730.404814] LNet: Added LNI 192.168.203.122@tcp106 [8/256/0/180] [13731.267774] LNet: Added LNI 192.168.203.122@tcp107 [8/256/0/180] [13732.212524] LNet: Added LNI 192.168.203.122@tcp108 [8/256/0/180] [13732.974486] LNet: Added LNI 192.168.203.122@tcp109 [8/256/0/180] [13733.691938] LNet: Added LNI 192.168.203.122@tcp110 [8/256/0/180] [13734.461806] LNet: Added LNI 192.168.203.122@tcp111 [8/256/0/180] [13735.276190] LNet: Added LNI 192.168.203.122@tcp112 [8/256/0/180] [13736.044554] LNet: Added LNI 192.168.203.122@tcp113 [8/256/0/180] [13736.878874] LNet: Added LNI 192.168.203.122@tcp114 [8/256/0/180] [13737.674701] LNet: Added LNI 192.168.203.122@tcp115 [8/256/0/180] [13738.599951] LNet: Added LNI 192.168.203.122@tcp116 [8/256/0/180] [13739.442043] LNet: Added LNI 192.168.203.122@tcp117 [8/256/0/180] [13740.356660] LNet: Added LNI 192.168.203.122@tcp118 [8/256/0/180] [13741.253404] LNet: Added LNI 192.168.203.122@tcp119 [8/256/0/180] [13742.040610] LNet: Added LNI 192.168.203.122@tcp120 [8/256/0/180] [13742.877726] LNet: Added LNI 192.168.203.122@tcp121 [8/256/0/180] [13743.694543] LNet: Added LNI 192.168.203.122@tcp122 [8/256/0/180] [13744.483047] LNet: Added LNI 192.168.203.122@tcp123 [8/256/0/180] [13745.251277] LNet: Added LNI 192.168.203.122@tcp124 [8/256/0/180] [13746.004366] LNet: Added LNI 192.168.203.122@tcp125 [8/256/0/180] [13746.820737] LNet: Added LNI 192.168.203.122@tcp126 [8/256/0/180] [13747.553334] LNet: Added LNI 192.168.203.122@tcp127 [8/256/0/180] [13748.252677] LNet: Added LNI 192.168.203.122@tcp128 [8/256/0/180] [13749.025672] LNet: Added LNI 192.168.203.122@tcp129 [8/256/0/180] [13749.807623] LNet: Added LNI 192.168.203.122@tcp130 [8/256/0/180] [13750.528363] LNet: Added LNI 192.168.203.122@tcp131 [8/256/0/180] [13751.297246] LNet: Added LNI 192.168.203.122@tcp132 [8/256/0/180] [13752.086209] LNet: Added LNI 192.168.203.122@tcp133 [8/256/0/180] [13752.826530] LNet: Added LNI 192.168.203.122@tcp134 [8/256/0/180] [13753.603675] LNet: Added LNI 192.168.203.122@tcp135 [8/256/0/180] [13759.015545] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13759.039551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13760.198854] Lustre: 463771:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.122@tcp: can't add NID 192.168.203.122@tcp130: rc = -75 [13760.268113] LNetError: 463798:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.122 [13760.280673] LNetError: Protocol error connecting to 192.168.203.122@tcp120 at host 192.168.203.122:988: is it running a compatible version of Lustre? [13760.294594] LNetError: Skipped 2 previous similar messages [13760.343483] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13760.368345] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13760.461885] Lustre: lustre-MDT0000: new disk, initializing [13760.531922] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13760.544321] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13761.248307] LNet: 4 local NIs in recovery (showing 4): 192.168.203.122@tcp123, 192.168.203.122@tcp124, 192.168.203.122@tcp125, 192.168.203.122@tcp120 [13761.253037] LNetError: 463792:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.122 [13761.293205] LNetError: 463792:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 5 previous similar messages [13761.312454] LNetError: Protocol error connecting to 192.168.203.122@tcp103 at host 192.168.203.122:988: is it running a compatible version of Lustre? [13761.342234] LNetError: Skipped 3 previous similar messages [13764.764412] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [13775.475556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13775.661526] Lustre: 463789:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13775.734636] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13775.742346] Lustre: Skipped 1 previous similar message [13775.916707] Lustre: lustre-MDT0001: new disk, initializing [13775.947538] Lustre: 464592:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.122@tcp: can't add NID 192.168.203.122@tcp130: rc = -75 [13775.957569] Lustre: 464592:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13776.112697] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13776.165637] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13776.176859] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13781.625613] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [13790.948357] LNetError: 463804:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.122 [13790.979476] LNetError: 463804:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [13790.997455] LNetError: Protocol error connecting to 192.168.203.122@tcp122 at host 192.168.203.122:988: is it running a compatible version of Lustre? [13792.114744] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13795.043845] LNetError: 461229:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.122 [13795.058457] LNetError: Protocol error connecting to 192.168.203.122@tcp123 at host 192.168.203.122:988: is it running a compatible version of Lustre? [13797.497387] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13800.165187] LNetError: 461231:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.122 [13800.178110] LNetError: Protocol error connecting to 192.168.203.122@tcp103 at host 192.168.203.122:988: is it running a compatible version of Lustre? [13803.420686] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13803.752328] Lustre: lustre-OST0000: new disk, initializing [13803.755506] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13803.761944] Lustre: 465763:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13803.832199] Lustre: 465768:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.122@tcp: can't add NID 192.168.203.122@tcp130: rc = -75 [13803.837168] Lustre: 465768:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13803.873629] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13808.353571] LNetError: 461230:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.122 [13808.361993] LNetError: 461230:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [13808.367550] LNetError: Protocol error connecting to 192.168.203.122@tcp123 at host 192.168.203.122:988: is it running a compatible version of Lustre? [13808.373346] LNetError: Skipped 1 previous similar message [13809.704531] Lustre: 466078:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.122@tcp: can't add NID 192.168.203.122@tcp130: rc = -75 [13809.712424] Lustre: 466078:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13809.807105] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13809.827210] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13809.989668] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13810.194629] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [13819.682409] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13825.765295] LNetError: 463792:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.122 [13825.786406] LNetError: 463792:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 3 previous similar messages [13825.805211] LNetError: Protocol error connecting to 192.168.203.122@tcp103 at host 192.168.203.122:988: is it running a compatible version of Lustre? [13825.820965] LNetError: Skipped 3 previous similar messages [13830.625881] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13830.639476] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13830.648134] LustreError: Skipped 1 previous similar message [13830.648977] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13830.669073] Lustre: Skipped 1 previous similar message [13835.754342] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13835.759930] Lustre: Skipped 1 previous similar message [13840.868630] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13840.875904] Lustre: Skipped 1 previous similar message [13841.376140] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13841.518146] Lustre: server umount lustre-OST0000 complete [13845.986334] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13846.002653] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13846.011543] Lustre: Skipped 2 previous similar messages [13851.114131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13851.130503] Lustre: Skipped 1 previous similar message [13851.469582] Lustre: server umount lustre-MDT0000 complete [13855.497307] LustreError: 463781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788587428 with bad export cookie 11978663261976576563 [13855.503924] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13855.512505] LustreError: 463781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13856.010957] Lustre: server umount lustre-MDT0001 complete [13859.555933] LNetError: 461231:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.122 [13859.569539] LNetError: 461231:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 4 previous similar messages [13859.576472] LNetError: Protocol error connecting to 192.168.203.122@tcp123 at host 192.168.203.122:988: is it running a compatible version of Lustre? [13859.584721] LNetError: Skipped 4 previous similar messages [13866.586407] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [13869.236315] Key type lgssc unregistered [13869.537955] LNet: 467678:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13869.552552] LNetError: 467678:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13869.557625] LNet: Removed LNI 192.168.203.122@tcp [13870.624649] LNet: Removed LNI 192.168.203.122@tcp100 [13870.634377] LNetError: 467678:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13870.641806] LNetError: 467678:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13870.647832] LNet: Removed LNI 192.168.203.122@tcp101 [13871.712164] LNet: Removed LNI 192.168.203.122@tcp102 [13871.720892] LNetError: 467678:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13871.746059] LNetError: 467678:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13871.768217] LNet: Removed LNI 192.168.203.122@tcp103 [13872.801360] LNet: Removed LNI 192.168.203.122@tcp104 [13873.824739] LNet: Removed LNI 192.168.203.122@tcp105 [13873.848456] LNetError: 467678:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13873.861788] LNetError: 467678:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [13873.878847] LNet: Removed LNI 192.168.203.122@tcp106 [13874.912185] LNet: Removed LNI 192.168.203.122@tcp107 [13874.923832] LNet: Removed LNI 192.168.203.122@tcp108 [13876.004075] LNet: Removed LNI 192.168.203.122@tcp109 [13877.026151] LNet: Removed LNI 192.168.203.122@tcp110 [13878.048493] LNet: Removed LNI 192.168.203.122@tcp111 [13878.058044] LNetError: 467678:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13878.073359] LNetError: 467678:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [13878.087481] LNet: Removed LNI 192.168.203.122@tcp112 [13879.136187] LNet: Removed LNI 192.168.203.122@tcp113 [13880.163159] LNet: Removed LNI 192.168.203.122@tcp114 [13881.184234] LNet: Removed LNI 192.168.203.122@tcp115 [13882.208186] LNet: Removed LNI 192.168.203.122@tcp116 [13883.232756] LNet: Removed LNI 192.168.203.122@tcp117 [13884.256159] LNet: Removed LNI 192.168.203.122@tcp118 [13885.282025] LNet: Removed LNI 192.168.203.122@tcp119 [13886.304181] LNet: Removed LNI 192.168.203.122@tcp120 [13886.308039] LNetError: 467678:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13886.314688] LNetError: 467678:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [13886.333246] LNet: Removed LNI 192.168.203.122@tcp121 [13887.392486] LNet: Removed LNI 192.168.203.122@tcp122 [13888.416225] LNet: Removed LNI 192.168.203.122@tcp123 [13889.441883] LNet: Removed LNI 192.168.203.122@tcp124 [13890.464236] LNet: Removed LNI 192.168.203.122@tcp125 [13891.489343] LNet: Removed LNI 192.168.203.122@tcp126 [13892.512166] LNet: Removed LNI 192.168.203.122@tcp127 [13893.536434] LNet: Removed LNI 192.168.203.122@tcp128 [13894.560155] LNet: Removed LNI 192.168.203.122@tcp129 [13894.567284] LNet: Removed LNI 192.168.203.122@tcp130 [13894.571808] LNet: Removed LNI 192.168.203.122@tcp131 [13894.574600] LNet: Removed LNI 192.168.203.122@tcp132 [13894.582812] LNet: Removed LNI 192.168.203.122@tcp133 [13894.588745] LNet: Removed LNI 192.168.203.122@tcp134 [13894.599747] LNet: Removed LNI 192.168.203.122@tcp135 [13895.582365] Key type .llcrypt unregistered [13895.585036] Key type ._llcrypt unregistered [13919.644575] Key type ._llcrypt registered [13919.647502] Key type .llcrypt registered [13919.790368] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [13934.485699] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [13935.549842] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13935.632349] alg: No test for adler32 (adler32-zlib) [13936.725647] Lustre: Lustre: Build Version: 2.17.58_4_g27fe0b7 [13936.942326] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [13938.648276] Key type lgssc registered [13939.691668] Lustre: Echo OBD driver; http://www.lustre.org/ [13975.159399] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 01:52:24 (1788587544) [13992.806655] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [13995.358289] Key type lgssc unregistered [13995.644151] LNet: 471760:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13995.650156] LNetError: 471760:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13995.662613] LNet: Removed LNI 192.168.203.122@tcp [13996.297542] Key type .llcrypt unregistered [13996.299324] Key type ._llcrypt unregistered [14011.459626] Key type ._llcrypt registered [14011.462856] Key type .llcrypt registered [14011.596823] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [14012.867335] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14012.897270] alg: No test for adler32 (adler32-zlib) [14013.914639] Lustre: Lustre: Build Version: 2.17.58_4_g27fe0b7 [14014.141462] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [14015.816377] Key type lgssc registered [14016.810931] Lustre: Echo OBD driver; http://www.lustre.org/ [14025.758906] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14025.815520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14027.071077] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14027.102222] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14027.182934] Lustre: lustre-MDT0000: new disk, initializing [14027.274062] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14027.290266] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14031.957401] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [14043.153732] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14043.386846] Lustre: 472969:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14043.493973] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14043.503334] Lustre: Skipped 1 previous similar message [14043.602899] Lustre: lustre-MDT0001: new disk, initializing [14043.698391] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14043.751318] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14043.773547] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14048.932938] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [14060.431778] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14067.024270] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14075.072146] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14075.591574] Lustre: lustre-OST0000: new disk, initializing [14075.599429] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14075.614853] Lustre: 474937:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14075.795112] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14082.259792] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [14084.172090] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14084.178601] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14084.263195] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14090.832595] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14094.821271] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing lnet_if_list [14099.778532] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14099.802072] LNet: Added LNI 192.168.203.122@tcp10 [8/256/0/180] [14105.881421] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14105.927840] LNet: Added LNI 192.168.203.122@tcp11 [8/256/0/180] [14110.109303] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14110.135882] LNet: Added LNI 192.168.203.122@tcp12 [8/256/0/180] [14114.487137] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14114.519103] LNet: Added LNI 192.168.203.122@tcp13 [8/256/0/180] [14118.290724] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14118.333492] LNet: Added LNI 192.168.203.122@tcp14 [8/256/0/180] [14122.281296] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14122.300327] LNet: Added LNI 192.168.203.122@tcp15 [8/256/0/180] [14127.660862] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14127.699856] LNet: Added LNI 192.168.203.122@tcp16 [8/256/0/180] [14133.668479] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14133.700861] LNet: Added LNI 192.168.203.122@tcp17 [8/256/0/180] [14138.733384] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14138.768036] LNet: Added LNI 192.168.203.122@tcp18 [8/256/0/180] [14143.695964] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14143.727931] LNet: Added LNI 192.168.203.122@tcp19 [8/256/0/180] [14149.737598] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14149.763348] LNet: Added LNI 192.168.203.122@tcp20 [8/256/0/180] [14155.664877] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14155.731630] LNet: Added LNI 192.168.203.122@tcp21 [8/256/0/180] [14162.900580] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14162.934590] LNet: Added LNI 192.168.203.122@tcp22 [8/256/0/180] [14168.766828] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14168.789598] LNet: Added LNI 192.168.203.122@tcp23 [8/256/0/180] [14176.050740] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14176.078303] LNet: Added LNI 192.168.203.122@tcp24 [8/256/0/180] [14182.148711] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14182.166578] LNet: Added LNI 192.168.203.122@tcp25 [8/256/0/180] [14188.300355] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14188.325387] LNet: Added LNI 192.168.203.122@tcp26 [8/256/0/180] [14194.692188] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14194.737548] LNet: Added LNI 192.168.203.122@tcp27 [8/256/0/180] [14201.907618] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14201.930760] LNet: Added LNI 192.168.203.122@tcp28 [8/256/0/180] [14209.396186] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14209.452390] LNet: Added LNI 192.168.203.122@tcp29 [8/256/0/180] [14215.198944] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14215.228412] LNet: Added LNI 192.168.203.122@tcp30 [8/256/0/180] [14221.543759] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14221.578448] LNet: Added LNI 192.168.203.122@tcp31 [8/256/0/180] [14228.109152] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14228.134566] LNet: Added LNI 192.168.203.122@tcp32 [8/256/0/180] [14233.916476] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14233.981216] LNet: Added LNI 192.168.203.122@tcp33 [8/256/0/180] [14240.685138] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14240.730829] LNet: Added LNI 192.168.203.122@tcp34 [8/256/0/180] [14247.113413] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14247.140497] LNet: Added LNI 192.168.203.122@tcp35 [8/256/0/180] [14253.163509] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14253.212355] LNet: Added LNI 192.168.203.122@tcp36 [8/256/0/180] [14259.309652] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14259.332607] LNet: Added LNI 192.168.203.122@tcp37 [8/256/0/180] [14264.435538] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14264.458242] LNet: Added LNI 192.168.203.122@tcp38 [8/256/0/180] [14270.541607] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14270.587142] LNet: Added LNI 192.168.203.122@tcp39 [8/256/0/180] [14276.940347] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14276.976495] LNet: Added LNI 192.168.203.122@tcp40 [8/256/0/180] [14284.098611] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14284.119486] LNet: Added LNI 192.168.203.122@tcp41 [8/256/0/180] [14291.644064] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [14291.694609] LNet: Added LNI 192.168.203.122@tcp42 [8/256/0/180] [14299.081836] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [14299.107458] LNet: Added LNI 192.168.203.122@tcp43 [8/256/0/180] [14306.014938] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [14306.039105] LNet: Added LNI 192.168.203.122@tcp44 [8/256/0/180] [14312.985305] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [14313.015692] LNet: Added LNI 192.168.203.122@tcp45 [8/256/0/180] [14320.903254] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [14320.982954] LNet: Added LNI 192.168.203.122@tcp46 [8/256/0/180] [14329.211774] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [14329.269223] LNet: Added LNI 192.168.203.122@tcp47 [8/256/0/180] [14338.634332] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [14338.663650] LNet: Added LNI 192.168.203.122@tcp48 [8/256/0/180] [14348.218264] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [14348.276636] LNet: Added LNI 192.168.203.122@tcp49 [8/256/0/180] [14359.351707] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [14359.412534] LNet: Added LNI 192.168.203.122@tcp50 [8/256/0/180] [14381.541092] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14381.543497] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14381.546985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14381.560109] Lustre: Skipped 1 previous similar message [14386.666373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14386.671523] Lustre: Skipped 4 previous similar messages [14391.780267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14394.337021] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14394.662655] Lustre: server umount lustre-MDT0000 complete [14398.472387] LustreError: 473773:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788587971 with bad export cookie 13448801182273765323 [14398.477194] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14398.487813] LustreError: 473773:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14398.808171] Lustre: server umount lustre-MDT0001 complete [14415.738635] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [14417.378347] Lustre: 472239:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788587974/real 1788587974] req@00000000a6cced81 x1875470017793408/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788587990 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14417.412822] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14418.464308] Lustre: 472239:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788587975/real 1788587975] req@000000005be98262 x1875470017793536/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788587991 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14423.524728] Lustre: 472241:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788587980/real 1788587980] req@000000009055ce34 x1875470017793664/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788587996 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14424.291764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14429.664365] Lustre: 472239:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788587986/real 1788587986] req@000000008aaf76e7 x1875470017794048/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788588002 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14429.689561] Lustre: 472239:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [14444.065016] LustreError: 472238:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000ead478c4 x1875470017794816/t0(0) o250->MGC192.168.203.122@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14444.302983] LustreError: 474930:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14444.317629] LustreError: 474930:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [14444.342923] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14448.166736] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [14448.360095] LustreError: 474931:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14451.426151] LustreError: 474930:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14451.428041] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14456.084412] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14456.490191] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14460.326529] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [14462.963081] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14466.918101] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14472.027646] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14488.559579] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14488.562719] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14488.586166] Lustre: Skipped 1 previous similar message [14488.601647] Lustre: Skipped 3 previous similar messages [14493.668715] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14493.675028] Lustre: Skipped 1 previous similar message [14494.721464] Lustre: server umount lustre-OST0000 complete [14498.786404] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14498.806693] Lustre: Skipped 1 previous similar message [14503.905221] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14503.918455] Lustre: Skipped 3 previous similar messages [14504.449872] Lustre: server umount lustre-MDT0000 complete [14508.337134] LustreError: 472960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788588081 with bad export cookie 13448801182273767990 [14508.347416] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14508.353406] LustreError: 472960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14508.653312] Lustre: server umount lustre-MDT0001 complete [14518.376745] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [14520.812342] Key type lgssc unregistered [14521.205338] LNet: 501974:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14521.210591] LNetError: 501974:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14521.218752] LNet: Removed LNI 192.168.203.122@tcp [14521.220785] LNet: Removed LNI 192.168.203.122@tcp10 [14521.222881] LNet: Removed LNI 192.168.203.122@tcp11 [14521.224561] LNet: Removed LNI 192.168.203.122@tcp12 [14521.225877] LNet: Removed LNI 192.168.203.122@tcp13 [14521.227092] LNet: Removed LNI 192.168.203.122@tcp14 [14521.229707] LNet: Removed LNI 192.168.203.122@tcp15 [14521.231843] LNet: Removed LNI 192.168.203.122@tcp16 [14521.233184] LNet: Removed LNI 192.168.203.122@tcp17 [14521.234507] LNet: Removed LNI 192.168.203.122@tcp18 [14521.236448] LNet: Removed LNI 192.168.203.122@tcp19 [14521.238258] LNet: Removed LNI 192.168.203.122@tcp20 [14521.239710] LNet: Removed LNI 192.168.203.122@tcp21 [14521.241172] LNet: Removed LNI 192.168.203.122@tcp22 [14521.242740] LNet: Removed LNI 192.168.203.122@tcp23 [14521.244235] LNet: Removed LNI 192.168.203.122@tcp24 [14521.245544] LNet: Removed LNI 192.168.203.122@tcp25 [14521.246974] LNet: Removed LNI 192.168.203.122@tcp26 [14521.248375] LNet: Removed LNI 192.168.203.122@tcp27 [14521.249732] LNet: Removed LNI 192.168.203.122@tcp28 [14521.251392] LNet: Removed LNI 192.168.203.122@tcp29 [14521.253693] LNet: Removed LNI 192.168.203.122@tcp30 [14521.255869] LNet: Removed LNI 192.168.203.122@tcp31 [14521.258286] LNet: Removed LNI 192.168.203.122@tcp32 [14521.259873] LNet: Removed LNI 192.168.203.122@tcp33 [14521.261714] LNet: Removed LNI 192.168.203.122@tcp34 [14521.263104] LNet: Removed LNI 192.168.203.122@tcp35 [14521.264445] LNet: Removed LNI 192.168.203.122@tcp36 [14521.265813] LNet: Removed LNI 192.168.203.122@tcp37 [14521.267322] LNet: Removed LNI 192.168.203.122@tcp38 [14521.268719] LNet: Removed LNI 192.168.203.122@tcp39 [14521.270225] LNet: Removed LNI 192.168.203.122@tcp40 [14521.271766] LNet: Removed LNI 192.168.203.122@tcp41 [14521.273617] LNet: Removed LNI 192.168.203.122@tcp42 [14521.279791] LNet: Removed LNI 192.168.203.122@tcp43 [14521.282278] LNet: Removed LNI 192.168.203.122@tcp44 [14521.284756] LNet: Removed LNI 192.168.203.122@tcp45 [14521.287362] LNet: Removed LNI 192.168.203.122@tcp46 [14521.292975] LNet: Removed LNI 192.168.203.122@tcp47 [14521.295314] LNet: Removed LNI 192.168.203.122@tcp48 [14521.298180] LNet: Removed LNI 192.168.203.122@tcp49 [14521.309964] LNet: Removed LNI 192.168.203.122@tcp50 [14522.093231] Key type .llcrypt unregistered [14522.094695] Key type ._llcrypt unregistered [14543.890431] Key type ._llcrypt registered [14543.892174] Key type .llcrypt registered [14543.994307] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [14559.508612] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [14560.848830] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14560.907266] alg: No test for adler32 (adler32-zlib) [14561.980752] Lustre: Lustre: Build Version: 2.17.58_4_g27fe0b7 [14562.148835] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [14563.800174] Key type lgssc registered [14564.893809] Lustre: Echo OBD driver; http://www.lustre.org/ [14598.466205] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14598.530994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14599.769944] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14599.803163] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14599.918925] Lustre: lustre-MDT0000: new disk, initializing [14600.016486] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14600.036654] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14604.793620] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [14614.929548] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14615.105441] Lustre: 505170:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14615.175943] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14615.180828] Lustre: Skipped 1 previous similar message [14615.257347] Lustre: lustre-MDT0001: new disk, initializing [14615.358532] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14615.390952] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14615.406137] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14619.893239] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [14629.752893] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14634.470705] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14640.422927] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14640.636419] Lustre: *** cfs_fail_loc=90f, val=0*** [14640.702345] Lustre: lustre-OST0000: new disk, initializing [14640.705324] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14640.711395] Lustre: 507185:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14640.778256] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14646.204600] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [14647.367087] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14647.385304] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14647.527735] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14654.185406] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14663.139860] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14663.140928] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14663.147708] LustreError: Skipped 1 previous similar message [14663.148482] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14663.174841] Lustre: Skipped 1 previous similar message [14668.024972] Lustre: server umount lustre-OST0000 complete [14671.841447] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14671.855523] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14671.879762] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14671.897613] Lustre: Skipped 1 previous similar message [14673.381520] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14673.390681] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14677.692419] Lustre: server umount lustre-MDT0000 complete [14681.477500] LustreError: 507184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788588254 with bad export cookie 8690281894097053422 [14681.478991] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14681.483887] LustreError: 507184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14681.919782] Lustre: server umount lustre-MDT0001 complete [14692.350191] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [14694.639634] Key type lgssc unregistered [14694.903774] LNet: 509088:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14694.909375] LNetError: 509088:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14694.920370] LNet: Removed LNI 192.168.203.122@tcp [14695.743302] Key type .llcrypt unregistered [14695.745472] Key type ._llcrypt unregistered [14714.856653] Key type ._llcrypt registered [14714.858517] Key type .llcrypt registered [14714.957705] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [14716.010779] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14716.038072] alg: No test for adler32 (adler32-zlib) [14717.288753] Lustre: Lustre: Build Version: 2.17.58_4_g27fe0b7 [14717.568261] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [14719.264166] Key type lgssc registered [14720.320923] Lustre: Echo OBD driver; http://www.lustre.org/ [14728.113057] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14728.130865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14729.686444] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14734.906504] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [14739.942804] LustreError: 510353:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14739.974684] LustreError: 510353:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [14744.534368] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14744.993068] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14749.837195] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [14757.850463] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14764.112702] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14770.674415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14771.022391] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14777.691737] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [14785.534644] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14799.859108] loop0: detected capacity change from 0 to 204800000 [14800.101108] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14800.118899] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14800.129786] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14806.771431] loop0: detected capacity change from 0 to 204800000 [14807.070409] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14807.088574] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14807.102359] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14812.571116] loop0: detected capacity change from 0 to 204800000 [14818.825856] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14819.023344] Lustre: fs15246-OST0000: new disk, initializing [14819.038410] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [14819.058199] Lustre: 513948:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14819.148330] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14825.867811] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [14836.118324] loop1: detected capacity change from 0 to 204800000 [14842.615878] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14842.749518] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [14842.812940] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [14842.917211] Lustre: fs15246-MDT0000: new disk, initializing [14843.107211] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14843.142558] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14843.239081] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [14844.258450] Lustre: 515072:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000022553485 x1875470754823552/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788588433 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [14848.130105] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [14849.321166] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [14849.332880] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [14849.378672] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [14874.617808] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14874.645019] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [14879.267518] Lustre: server umount fs15246-MDT0000 complete [14879.744297] LustreError: 512098:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14884.844729] LustreError: 510348:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14888.602333] Lustre: server umount fs15246-OST0000 complete [14902.251151] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14902.252229] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14902.265632] LustreError: Skipped 1 previous similar message [14902.267285] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14902.324413] Lustre: Skipped 1 previous similar message [14905.316329] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14905.331780] Lustre: Skipped 2 previous similar messages [14906.921102] Lustre: server umount lustre-OST0000 complete [14914.019780] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14914.037621] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14914.060336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14917.100221] Lustre: server umount lustre-MDT0000 complete [14920.673938] LustreError: 510348:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14920.703107] LustreError: 510348:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14921.138317] LustreError: 510334:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788588494 with bad export cookie 15569098458156803075 [14921.141139] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14921.158174] LustreError: 510334:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14921.475696] Lustre: server umount lustre-MDT0001 complete [14931.386760] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [14934.592104] Key type lgssc unregistered [14934.857839] LNet: 518169:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14934.866747] LNetError: 518169:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14934.881355] LNet: Removed LNI 192.168.203.122@tcp [14936.148774] Key type .llcrypt unregistered [14936.154651] Key type ._llcrypt unregistered [14957.567803] Key type ._llcrypt registered [14957.573300] Key type .llcrypt registered [14957.705703] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [14958.729901] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14958.807784] alg: No test for adler32 (adler32-zlib) [14959.824649] Lustre: Lustre: Build Version: 2.17.58_4_g27fe0b7 [14960.025327] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [14961.689606] Key type lgssc registered [14962.860440] Lustre: Echo OBD driver; http://www.lustre.org/ [15003.931804] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [15014.873353] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [15051.590366] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15051.607581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15052.834441] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15052.888194] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15052.955991] Lustre: lustre-MDT0000: new disk, initializing [15053.008675] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15053.025795] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15060.092651] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [15072.333641] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15072.540607] Lustre: 522636:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15072.645420] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15072.653886] Lustre: Skipped 1 previous similar message [15072.734880] Lustre: lustre-MDT0001: new disk, initializing [15072.976419] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15073.033587] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15073.061880] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15078.202611] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [15087.691413] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15093.822210] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15101.561597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15102.060222] Lustre: lustre-OST0000: new disk, initializing [15102.071955] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15102.091382] Lustre: 524604:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15102.261852] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15109.190091] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15109.207569] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15109.341869] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15109.421191] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [15119.970471] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15125.353964] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15125.602138] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15130.105906] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15130.283205] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15135.204230] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15135.209982] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15135.234599] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15135.260309] Lustre: Skipped 1 previous similar message [15139.010130] Lustre: server umount lustre-OST0000 complete [15144.936498] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15144.946305] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15144.957950] Lustre: Skipped 1 previous similar message [15144.971846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15150.562208] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15150.577124] Lustre: Skipped 3 previous similar messages [15155.682557] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15155.689796] Lustre: Skipped 1 previous similar message [15157.216522] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15157.628846] Lustre: server umount lustre-MDT0000 complete [15162.756911] LustreError: 522627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788588735 with bad export cookie 2368385767426847478 [15162.769264] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15162.775294] LustreError: 522627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15163.410381] Lustre: server umount lustre-MDT0001 complete [15191.692314] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [15201.797660] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [15233.969219] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14911 sec ========= 02:13:23 (1788588803) [15235.859750] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 02:13:25 (1788588805) === [15239.687605] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 02:13:29 (1788588809) === [15282.480562] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [15285.283290] Key type lgssc unregistered [15285.628603] LNet: 531249:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15285.642391] LNetError: 531249:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15285.662244] LNet: Removed LNI 192.168.203.122@tcp [15286.772616] Key type .llcrypt unregistered [15286.775397] Key type ._llcrypt unregistered