[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 525035621 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003153] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005013] kvm-guest: setup PV IPIs [ 0.008503] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009026] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010012] pid_max: default: 32768 minimum: 301 [ 0.011138] LSM: Security Framework initializing [ 0.012057] Yama: becoming mindful. [ 0.013034] SELinux: Initializing. [ 0.015036] *** VALIDATE selinux *** [ 0.023369] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027745] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028151] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029104] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030110] *** VALIDATE tmpfs *** [ 0.032318] *** VALIDATE proc *** [ 0.033250] *** VALIDATE cgroup *** [ 0.034010] *** VALIDATE cgroup2 *** [ 0.035253] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036157] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038027] Spectre V2 : User space: Vulnerable [ 0.039008] Speculative Store Bypass: Vulnerable [ 0.042386] debug: unmapping init [mem 0xffffffff9fe59000-0xffffffff9fe60fff] [ 0.044911] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045686] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046024] ... version: 2 [ 0.047014] ... bit width: 48 [ 0.048011] ... generic registers: 4 [ 0.049014] ... value mask: 0000ffffffffffff [ 0.050014] ... max period: 00007fffffffffff [ 0.051014] ... fixed-purpose events: 3 [ 0.052011] ... event mask: 000000070000000f [ 0.053296] rcu: Hierarchical SRCU implementation. [ 0.055536] smp: Bringing up secondary CPUs ... [ 0.056579] x86: Booting SMP configuration: [ 0.057024] .... node #0, CPUs: #1 #2 #3 [ 0.060390] smp: Brought up 1 node, 4 CPUs [ 0.062013] smpboot: Max logical packages: 1 [ 0.063015] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.141096] node 0 deferred pages initialised in 76ms [ 0.145009] devtmpfs: initialized [ 0.146285] x86/mm: Memory block size: 128MB [ 0.148981] gcov: version magic: 0x41383552 [ 0.152316] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.153076] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.154296] pinctrl core: initialized pinctrl subsystem [ 0.155178] [ 0.155811] ************************************************************* [ 0.156014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.157012] ** ** [ 0.158012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.159011] ** ** [ 0.160010] ** This means that this kernel is built to expose internal ** [ 0.161012] ** IOMMU data structures, which may compromise security on ** [ 0.162079] ** your system. ** [ 0.163013] ** ** [ 0.164010] ** If you see this message and you are not debugging the ** [ 0.165011] ** kernel, report this immediately to your vendor! ** [ 0.166009] ** ** [ 0.167014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.168012] ************************************************************* [ 0.169905] NET: Registered protocol family 16 [ 0.171462] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.174076] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.178066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.181127] cpuidle: using governor menu [ 0.185207] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.188712] PCI: Using configuration type 1 for base access [ 0.191293] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.203054] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.205049] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.208085] cryptd: max_cpu_qlen set to 1000 [ 0.212254] ACPI: Added _OSI(Module Device) [ 0.214017] ACPI: Added _OSI(Processor Device) [ 0.215010] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.216010] ACPI: Added _OSI(Processor Aggregator Device) [ 0.221471] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.227100] ACPI: Interpreter enabled [ 0.228058] ACPI: PM: (supports S0 S3 S4 S5) [ 0.230012] ACPI: Using IOAPIC for interrupt routing [ 0.232120] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.235387] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.245942] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.248077] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.250020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.253131] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.257033] acpiphp: Slot [2] registered [ 0.258029] acpiphp: Slot [5] registered [ 0.258984] acpiphp: Slot [6] registered [ 0.260083] acpiphp: Slot [7] registered [ 0.261040] acpiphp: Slot [8] registered [ 0.262018] acpiphp: Slot [9] registered [ 0.263019] acpiphp: Slot [10] registered [ 0.264019] acpiphp: Slot [3] registered [ 0.264974] acpiphp: Slot [4] registered [ 0.266102] acpiphp: Slot [11] registered [ 0.267089] acpiphp: Slot [12] registered [ 0.269118] acpiphp: Slot [13] registered [ 0.270161] acpiphp: Slot [14] registered [ 0.272079] acpiphp: Slot [15] registered [ 0.273093] acpiphp: Slot [16] registered [ 0.274066] acpiphp: Slot [17] registered [ 0.274998] acpiphp: Slot [18] registered [ 0.275076] acpiphp: Slot [19] registered [ 0.277068] acpiphp: Slot [20] registered [ 0.278082] acpiphp: Slot [21] registered [ 0.279085] acpiphp: Slot [22] registered [ 0.280060] acpiphp: Slot [23] registered [ 0.281116] acpiphp: Slot [24] registered [ 0.283063] acpiphp: Slot [25] registered [ 0.284069] acpiphp: Slot [26] registered [ 0.285045] acpiphp: Slot [27] registered [ 0.286070] acpiphp: Slot [28] registered [ 0.287000] acpiphp: Slot [29] registered [ 0.287000] acpiphp: Slot [30] registered [ 0.289096] acpiphp: Slot [31] registered [ 0.290064] PCI host bridge to bus 0000:00 [ 0.292021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.294020] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.296028] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.299032] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.301026] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.304026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.306183] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.309082] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.312354] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.322015] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.326514] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.329026] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.331016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.333016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.335610] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.339086] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.341033] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.343793] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.349014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.362959] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.366983] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.373061] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.380018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.391015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.407018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.421570] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.435015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.445017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.463016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.475389] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.486017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.495255] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.512017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.522971] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.533023] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.540018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.559016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.569083] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.575018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.582016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.602016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.611451] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.618012] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.625020] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.649019] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.661987] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.664487] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.667314] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.670401] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.672403] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.676106] iommu: Default domain type: Passthrough [ 0.677581] SCSI subsystem initialized [ 0.679121] ACPI: bus type USB registered [ 0.680099] usbcore: registered new interface driver usbfs [ 0.682135] usbcore: registered new interface driver hub [ 0.684098] usbcore: registered new device driver usb [ 0.686157] pps_core: LinuxPPS API ver. 1 registered [ 0.688010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.691065] PTP clock support registered [ 0.693131] EDAC MC: Ver: 3.0.0 [ 0.695491] PCI: Using ACPI for IRQ routing [ 0.697830] NetLabel: Initializing [ 0.699012] NetLabel: domain hash size = 128 [ 0.700011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.702085] NetLabel: unlabeled traffic allowed by default [ 0.705097] vgaarb: loaded [ 0.706253] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.708012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.713300] clocksource: Switched to clocksource kvm-clock [ 0.852141] VFS: Disk quotas dquot_6.6.0 [ 0.854175] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.857499] *** VALIDATE ramfs *** [ 0.858645] *** VALIDATE hugetlbfs *** [ 0.861315] pnp: PnP ACPI init [ 0.863834] pnp: PnP ACPI: found 6 devices [ 0.882646] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.885895] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.888117] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.890141] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.892563] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.895079] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.897995] NET: Registered protocol family 2 [ 0.900505] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.904058] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.907492] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.913161] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.916739] TCP: Hash tables configured (established 65536 bind 65536) [ 0.919788] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.922927] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.925904] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.928992] NET: Registered protocol family 1 [ 0.931849] RPC: Registered named UNIX socket transport module. [ 0.934159] RPC: Registered udp transport module. [ 0.935702] RPC: Registered tcp transport module. [ 0.937045] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.939402] NET: Registered protocol family 44 [ 0.941033] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.943243] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.945268] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.947701] PCI: CLS 0 bytes, default 64 [ 0.950090] Unpacking initramfs... [ 2.362323] debug: unmapping init [mem 0xffff988bbcc54000-0xffff988bbffbffff] [ 2.366328] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.367857] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.370468] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.858400] Initialise system trusted keyrings [ 2.859723] Key type blacklist registered [ 2.861416] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.875385] zbud: loaded [ 2.879647] *** VALIDATE nfs *** [ 2.881025] *** VALIDATE nfs4 *** [ 2.882941] pstore: using deflate compression [ 2.888520] Platform Keyring initialized [ 2.997332] NET: Registered protocol family 38 [ 2.999052] Key type asymmetric registered [ 2.999849] Asymmetric key parser 'x509' registered [ 3.001210] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.003489] io scheduler mq-deadline registered [ 3.004869] io scheduler kyber registered [ 3.006010] io scheduler bfq registered [ 3.007164] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.009221] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.012276] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.015330] ACPI: Power Button [PWRF] [ 3.019820] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.027230] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.038676] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.045879] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.065803] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.095625] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.124374] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.129384] Non-volatile memory driver v1.3 [ 3.130905] Linux agpgart interface v0.103 [ 3.167382] virtio_blk virtio1: [vda] 149768 512-byte logical blocks (76.7 MB/73.1 MiB) [ 3.170508] vda: detected capacity change from 0 to 76681216 [ 3.186957] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.190114] vdb: detected capacity change from 0 to 1073741824 [ 3.205070] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.207872] vdc: detected capacity change from 0 to 2621440000 [ 3.225916] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.228707] vdd: detected capacity change from 0 to 2621440000 [ 3.250889] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.254340] vde: detected capacity change from 0 to 4294967296 [ 3.269166] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.272195] vdf: detected capacity change from 0 to 4294967296 [ 3.283614] libphy: Fixed MDIO Bus: probed [ 3.291622] usbcore: registered new interface driver usbserial_generic [ 3.294144] usbserial: USB Serial support registered for generic [ 3.296479] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.300781] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.302775] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.305678] mousedev: PS/2 mouse device common for all mice [ 3.309522] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.310364] rtc_cmos 00:05: RTC can wake from S4 [ 3.316312] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.317783] rtc_cmos 00:05: registered as rtc0 [ 3.321485] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.324399] intel_pstate: CPU model not supported [ 3.324533] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.330619] hid: raw HID events driver (C) Jiri Kosina [ 3.332803] usbcore: registered new interface driver usbhid [ 3.335247] usbhid: USB HID core driver [ 3.336836] drop_monitor: Initializing network drop monitor service [ 3.339120] Initializing XFRM netlink socket [ 3.341123] NET: Registered protocol family 10 [ 3.343953] Segment Routing with IPv6 [ 3.345332] NET: Registered protocol family 17 [ 3.347151] mpls_gso: MPLS GSO support [ 3.353478] RAS: Correctable Errors collector initialized. [ 3.355978] AVX version of gcm_enc/dec engaged. [ 3.358314] AES CTR mode by8 optimization enabled [ 3.433055] sched_clock: Marking stable (3433012135, 0)->(4384328120, -951315985) [ 3.437238] registered taskstats version 1 [ 3.439912] Loading compiled-in X.509 certificates [ 3.441346] zswap: loaded using pool lzo/zbud [ 3.460666] Key type big_key registered [ 3.472069] Key type encrypted registered [ 3.473947] ima: No TPM chip found, activating TPM-bypass! [ 3.475762] ima: Allocated hash algorithm: sha1 [ 3.477381] ima: No architecture policies found [ 3.478975] evm: Initialising EVM extended attributes: [ 3.480873] evm: security.selinux [ 3.482010] evm: security.ima [ 3.483167] evm: security.capability [ 3.484434] evm: HMAC attrs: 0x1 [ 3.486871] rtc_cmos 00:05: setting system clock to 2026-09-05 16:06:10 UTC (1788624370) [ 3.492792] debug: unmapping init [mem 0xffffffffa0e03000-0xffffffffa0ffffff] [ 3.495683] debug: unmapping init [mem 0xffffffff9fb82000-0xffffffff9fe58fff] [ 3.504104] Write protecting the kernel read-only data: 28672k [ 3.507427] debug: unmapping init [mem 0xffffffff9e203000-0xffffffff9e3fffff] [ 3.509921] debug: unmapping init [mem 0xffffffff9eb14000-0xffffffff9ebfffff] [ 3.546097] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.554860] systemd[1]: Detected virtualization kvm. [ 3.556779] systemd[1]: Detected architecture x86-64. [ 3.558953] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.588578] systemd[1]: No hostname configured. [ 3.590429] systemd[1]: Set hostname to . [ 3.592374] random: systemd: uninitialized urandom read (16 bytes read) [ 3.594653] systemd[1]: Initializing machine ID from random generator. [ 3.634705] random: ln: uninitialized urandom read (6 bytes read) [ 3.709123] random: systemd: uninitialized urandom read (16 bytes read) [ 3.710880] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.714203] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.718219] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Reached target Slices. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Journal Socket (/dev/log). Starting Apply Kernel Variables... Starting Journal Service... [ OK ] Listening on udev Control Socket. Starting Setup Virtual Console... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.226544] device-mapper: uevent: version 1.0.3 [ 4.229256] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.869477] virtio_net virtio0 ens2: renamed from eth0 [ 4.882925] random: fast init done [ 4.955399] scsi host0: ata_piix [ 4.976318] scsi host1: ata_piix [ 4.978318] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.980872] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.600798] dracut-initqueue[580]: RTNETLINK answers: File exists [ 9.990248] random: crng init done [ 9.991402] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.295848] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.382152] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.630826] SELinux: Disabled at runtime. [ 11.692303] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.699230] systemd[1]: Detected virtualization kvm. [ 11.700452] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.216033] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.219567] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.223773] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.226958] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.230118] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.236302] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.247684] systemd[1]: Mounting POSIX Message Queue File System... Mounting POSIX Message Queue File System... Starting Create list of required st…ce nodes for the current kernel... Mounting Kernel Debug File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Kernel Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target rpc_pipefs.target. Starting Remount Root and Kernel File Systems... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Stopped target Initrd File Systems. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target RPC Port Mapper. Starting Apply Kernel Variables... Starting udev Coldplug all Devices... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted[ 12.351951] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Volumes. [ OK ] Reached target Paths. [ OK ] Listening on Process Core Dump Socket. Mounting Huge Pages File System... [ OK ] Created slice system-getty.slice. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.669876] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.928781] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.057593] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.083408] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.098325] EDAC sbridge: Ver: 1.1.2 [ 14.655000] Key type dns_resolver registered [ 14.950103] NFS: Registering the id_resolver key type [ 14.951716] Key type id_resolver registered [ 14.953260] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started OpenSSH server daemon. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg227-server login: [ 44.099887] hrtimer: interrupt took 7530474 ns [ 80.448670] libcfs: loading out-of-tree module taints kernel. [ 80.502618] Key type ._llcrypt registered [ 80.504871] Key type .llcrypt registered [ 80.596118] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_hostid [ 99.418714] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 101.005733] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 101.018318] alg: No test for adler32 (adler32-zlib) [ 102.470668] Lustre: Lustre: Build Version: 2.17.58_2_g69c2166 [ 103.256315] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 104.977790] Key type lgssc registered [ 106.661386] Lustre: Echo OBD driver; http://www.lustre.org/ [ 124.325227] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 163.527972] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 177.167581] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 177.215525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 178.373952] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 178.404090] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 178.478680] Lustre: lustre-MDT0000: new disk, initializing [ 178.578934] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 178.625728] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 183.960827] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 203.479746] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 203.647914] Lustre: 6515:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 203.726804] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 203.748296] Lustre: Skipped 1 previous similar message [ 203.906641] Lustre: lustre-MDT0001: new disk, initializing [ 204.097840] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 204.149200] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 204.166882] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 211.909706] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 217.397677] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 229.911578] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 230.253553] Lustre: lustre-OST0000: new disk, initializing [ 230.258542] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 230.265379] Lustre: 8454:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 230.342597] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 236.073099] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 236.087259] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 236.190414] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 237.659748] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 255.729273] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 255.895454] Lustre: lustre-OST0001: new disk, initializing [ 255.899923] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 255.908232] Lustre: 9530:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 255.971202] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 262.272570] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 262.279162] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 262.363771] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 263.463925] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 276.425574] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 284.151315] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 290.820493] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing check_logdir /tmp/testlogs/ [ 296.995634] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing yml_node [ 300.571577] Lustre: DEBUG MARKER: Client: 2.17.58.2 [ 303.281689] Lustre: DEBUG MARKER: MDS: 2.17.58.2 [ 305.856687] Lustre: DEBUG MARKER: OSS: 2.17.58.2 [ 307.975479] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Sep 5 12:11:13 EDT 2026 [ 331.313596] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 333.430199] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 349.667203] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 349.672653] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 349.680138] Lustre: Skipped 1 previous similar message [ 354.787890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 354.793536] Lustre: Skipped 6 previous similar messages [ 355.612943] Lustre: server umount lustre-MDT0000 complete [ 359.957491] LustreError: 6508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788624726 with bad export cookie 8511054262467857199 [ 359.958924] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 359.961698] LustreError: 6508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 365.026538] LustreError: 6522:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 365.026864] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 365.027623] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 365.061327] LustreError: 6522:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 365.078606] Lustre: Skipped 3 previous similar messages [ 369.186134] LustreError: 9781:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 369.186409] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 369.201160] LustreError: 9781:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 369.216733] Lustre: Skipped 2 previous similar messages [ 374.241350] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 374.243311] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 374.243403] LustreError: 6522:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 374.286860] Lustre: Skipped 1 previous similar message [ 374.799911] Lustre: server umount lustre-MDT0001 complete [ 379.664480] Lustre: server umount lustre-OST0000 complete [ 383.664451] Lustre: server umount lustre-OST0001 complete [ 391.635127] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_hostid [ 403.407167] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 450.582692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 450.905345] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 450.987243] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 451.104458] Lustre: lustre-MDT0000: new disk, initializing [ 451.237170] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 451.261382] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 457.230495] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 468.852550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 469.012357] Lustre: 15092:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 469.086445] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 469.091181] Lustre: Skipped 1 previous similar message [ 469.184840] Lustre: lustre-MDT0001: new disk, initializing [ 469.298573] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 469.374171] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 469.396516] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 474.804269] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 485.885216] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 492.150226] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 499.484667] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 499.739622] Lustre: lustre-OST0000: new disk, initializing [ 499.749138] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 499.755613] Lustre: 17063:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 499.832838] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 500.939636] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 500.953612] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 501.164963] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 507.545797] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 516.570792] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 521.375409] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 521.625254] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 526.421928] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 526.635415] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 531.940617] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 531.945134] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 531.962981] Lustre: Skipped 1 previous similar message [ 531.978169] Lustre: Skipped 1 previous similar message [ 534.628707] Lustre: server umount lustre-OST0000 complete [ 541.153817] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 541.168333] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 545.195878] Lustre: server umount lustre-MDT0000 complete [ 547.297854] LustreError: 15097:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 547.318073] LustreError: 15097:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 549.018802] LustreError: 15085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788624916 with bad export cookie 8511054262467858732 [ 549.020480] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 549.027102] LustreError: 15085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 549.350600] Lustre: server umount lustre-MDT0001 complete [ 553.703926] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 12:15:19 (1788624919) [ 555.909349] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 557.834121] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 12:15:23 (1788624923) [ 565.158418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 565.783209] LustreError: 19471:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 565.877343] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 571.570921] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 575.459736] LustreError: 19471:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 575.473898] LustreError: 19471:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 582.667890] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 588.071201] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 596.954295] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 602.948909] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 610.415673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 618.415381] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 625.795065] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 638.438567] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 638.445842] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 638.448747] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 638.452617] Lustre: Skipped 2 previous similar messages [ 638.461864] Lustre: Skipped 4 previous similar messages [ 643.564223] LustreError: 21294:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 643.590137] LustreError: 21294:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 643.791685] Lustre: server umount lustre-OST0000 complete [ 648.674531] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 648.712823] Lustre: Skipped 1 previous similar message [ 654.343030] Lustre: server umount lustre-MDT0000 complete [ 659.008781] LustreError: 19452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788625026 with bad export cookie 8511054262467859614 [ 659.014681] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 659.019621] LustreError: 19452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 659.342360] Lustre: server umount lustre-MDT0001 complete [ 671.962534] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 675.566523] Key type lgssc unregistered [ 676.015093] LNet: 23007:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 676.031657] LNetError: 23007:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 676.056856] LNet: Removed LNI 192.168.202.127@tcp [ 677.474170] Key type .llcrypt unregistered [ 677.481891] Key type ._llcrypt unregistered [ 702.788711] Key type ._llcrypt registered [ 702.791047] Key type .llcrypt registered [ 702.887736] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 704.011779] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 704.111321] alg: No test for adler32 (adler32-zlib) [ 705.326361] Lustre: Lustre: Build Version: 2.17.58_2_g69c2166 [ 705.541971] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 707.200197] Key type lgssc registered [ 708.462097] Lustre: Echo OBD driver; http://www.lustre.org/ [ 717.105113] loop: module loaded [ 717.194208] loop0: detected capacity change from 0 to 204800000 [ 717.313449] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 717.326776] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 717.503355] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 723.483426] loop0: detected capacity change from 0 to 204800000 [ 723.660510] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 723.671135] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 723.684424] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 729.360673] loop0: detected capacity change from 0 to 204800000 [ 736.619772] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 736.646950] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 737.930371] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 738.008609] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 738.171783] Lustre: lustre-MDT0000: new disk, initializing [ 738.268585] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 738.287927] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 743.847581] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 754.225167] loop1: detected capacity change from 0 to 204800000 [ 761.322668] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 761.843550] Lustre: lustre-OST07c6: new disk, initializing [ 761.847823] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 761.851630] Lustre: Skipped 1 previous similar message [ 761.854750] Lustre: 26190:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 762.021212] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 769.327793] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 771.722385] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 771.732992] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 771.766330] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 775.357151] Lustre: 25179:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 255 < left 264, rollback = 2 [ 775.370924] Lustre: 25179:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 775.387507] Lustre: 25179:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 775.401368] Lustre: 25179:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 775.409380] Lustre: 25179:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 775.417472] Lustre: 25179:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 788.712037] Lustre: 25178:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 788.724126] Lustre: 25178:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 788.737372] Lustre: 25178:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 788.747526] Lustre: 25178:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 788.759563] Lustre: 25178:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 788.772792] Lustre: 25178:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 794.388208] Lustre: 26843:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 794.401856] Lustre: 26843:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 794.412423] Lustre: 26843:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 794.423042] Lustre: 26843:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 794.429115] Lustre: 26843:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 794.440413] Lustre: 26843:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 794.456912] Lustre: 26843:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 794.467142] Lustre: 26843:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 794.480566] Lustre: 26843:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 794.492849] Lustre: 26843:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 794.499495] Lustre: 26843:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 794.516120] Lustre: 26843:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 799.787284] Lustre: 25177:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 799.810192] Lustre: 25177:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 799.825675] Lustre: 25177:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 799.834045] Lustre: 25177:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 799.841303] Lustre: 25177:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 799.851563] Lustre: 25177:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 799.865498] Lustre: 25177:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 799.882000] Lustre: 25177:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 799.894101] Lustre: 25177:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 799.903344] Lustre: 25177:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 799.913968] Lustre: 25177:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 799.925368] Lustre: 25177:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 806.525800] Lustre: 25179:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 806.537750] Lustre: 25179:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 806.545562] Lustre: 25179:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 806.551975] Lustre: 25179:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 806.565881] Lustre: 25179:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 806.572036] Lustre: 25179:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 806.584823] Lustre: 25179:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 806.592553] Lustre: 25179:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 806.607228] Lustre: 25179:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 806.617439] Lustre: 25179:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 806.625968] Lustre: 25179:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 806.631869] Lustre: 25179:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 815.131293] Lustre: 25179:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 815.151027] Lustre: 25179:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 815.158357] Lustre: 25179:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 815.167852] Lustre: 25179:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 815.176392] Lustre: 25179:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 815.187415] Lustre: 25179:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 815.198232] Lustre: 25179:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 815.207841] Lustre: 25179:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 815.220049] Lustre: 25179:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 815.230826] Lustre: 25179:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 815.237862] Lustre: 25179:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 815.246904] Lustre: 25179:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 863.713964] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 863.728986] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 868.837567] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 869.678810] Lustre: server umount lustre-OST07c6 complete [ 879.536952] Lustre: server umount lustre-MDT0000 complete [ 896.531904] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 899.446242] Key type lgssc unregistered [ 899.753564] LNet: 28743:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 899.770687] LNetError: 28743:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 899.817306] LNet: Removed LNI 192.168.202.127@tcp [ 900.968457] Key type .llcrypt unregistered [ 900.970829] Key type ._llcrypt unregistered [ 923.401255] Key type ._llcrypt registered [ 923.403310] Key type .llcrypt registered [ 923.527345] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 924.459600] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 924.535828] alg: No test for adler32 (adler32-zlib) [ 925.658765] Lustre: Lustre: Build Version: 2.17.58_2_g69c2166 [ 925.952749] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 927.648243] Key type lgssc registered [ 928.921287] Lustre: Echo OBD driver; http://www.lustre.org/ [ 937.469829] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 937.502971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 938.878723] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 944.126214] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 948.197974] LustreError: 30011:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 948.215348] LustreError: 30011:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 952.306722] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 952.502731] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 956.859535] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 965.193415] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 973.361600] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 982.100681] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 982.468989] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 983.566171] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 989.170401] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 996.808692] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1014.246618] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1014.256921] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1014.282465] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1015.521305] Lustre: server umount lustre-OST0000 complete [ 1021.412352] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1021.415053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1021.428658] Lustre: Skipped 1 previous similar message [ 1021.440509] Lustre: Skipped 1 previous similar message [ 1026.057181] Lustre: server umount lustre-MDT0000 complete [ 1026.541887] LustreError: 30008:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1026.560667] LustreError: 30008:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1029.681701] LustreError: 31859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788625396 with bad export cookie 15791088898633770392 [ 1029.683914] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1029.702177] LustreError: 31859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1039.498583] Lustre: server umount lustre-MDT0001 complete [ 1053.079257] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 1056.288608] Key type lgssc unregistered [ 1056.640820] LNet: 33748:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1056.655856] LNetError: 33748:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1056.690298] LNet: Removed LNI 192.168.202.127@tcp [ 1057.583077] Key type .llcrypt unregistered [ 1057.585282] Key type ._llcrypt unregistered [ 1076.860760] Key type ._llcrypt registered [ 1076.863498] Key type .llcrypt registered [ 1076.954506] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 1078.317847] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1078.333275] alg: No test for adler32 (adler32-zlib) [ 1079.374471] Lustre: Lustre: Build Version: 2.17.58_2_g69c2166 [ 1079.695597] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 1081.425936] Key type lgssc registered [ 1082.578466] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1091.138857] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1091.149799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1092.634557] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1097.521934] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 1102.823291] LustreError: 35014:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1102.834511] LustreError: 35014:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1106.965588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1107.388575] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1112.625883] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 1121.676395] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1128.443173] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1136.449703] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1136.904694] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1137.937953] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1143.346110] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 1152.595823] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1167.865802] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1167.880610] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1167.887048] Lustre: Skipped 1 previous similar message [ 1167.891259] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1167.930371] Lustre: Skipped 1 previous similar message [ 1172.102896] Lustre: server umount lustre-OST0000 complete [ 1178.083834] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1178.090416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1178.096414] Lustre: Skipped 1 previous similar message [ 1178.111789] Lustre: Skipped 1 previous similar message [ 1183.205286] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1184.217808] Lustre: server umount lustre-MDT0000 complete [ 1187.318183] LustreError: 35010:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1187.370685] LustreError: 35010:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1190.910505] LustreError: 34994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788625557 with bad export cookie 13543879927345477477 [ 1190.912492] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1190.922414] LustreError: 34994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1192.164309] Lustre: server umount lustre-MDT0001 complete [ 1205.732169] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 1207.909956] Key type lgssc unregistered [ 1208.091668] LNet: 38569:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1208.099277] LNetError: 38569:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1208.117409] LNet: Removed LNI 192.168.202.127@tcp [ 1208.775186] Key type .llcrypt unregistered [ 1208.779444] Key type ._llcrypt unregistered [ 1236.562745] Key type ._llcrypt registered [ 1236.565393] Key type .llcrypt registered [ 1236.751561] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 1238.095539] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1238.195272] alg: No test for adler32 (adler32-zlib) [ 1239.309732] Lustre: Lustre: Build Version: 2.17.58_2_g69c2166 [ 1239.637457] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 1241.336223] Key type lgssc registered [ 1242.873972] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1251.752339] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1251.785388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1253.280863] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1258.755132] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 1263.602031] LustreError: 39983:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1263.619797] LustreError: 39983:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1268.716338] LustreError: 39984:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1269.770847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1270.411471] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1276.576704] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 1285.660866] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1291.812938] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1299.229927] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1299.619959] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1306.701661] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 1316.628423] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1323.176295] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1333.730959] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1333.737649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1333.755183] Lustre: Skipped 1 previous similar message [ 1333.778439] Lustre: Skipped 2 previous similar messages [ 1337.851668] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.27@tcp (stopping) [ 1339.544505] Lustre: server umount lustre-MDT0000 complete [ 1342.940416] LustreError: 39979:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1344.108720] LustreError: 39965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788625711 with bad export cookie 9788795031392670830 [ 1344.119307] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1344.127339] LustreError: 39965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1348.042914] LustreError: 39980:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1348.061665] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.27@tcp (stopping) [ 1348.069306] LustreError: 39980:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1348.084305] Lustre: Skipped 3 previous similar messages [ 1349.090959] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1349.101911] Lustre: Skipped 1 previous similar message [ 1350.681343] Lustre: server umount lustre-MDT0001 complete [ 1367.009316] Lustre: server umount lustre-OST0000 complete [ 1386.030887] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 1389.278962] Key type lgssc unregistered [ 1389.610678] LNet: 43865:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1389.624613] LNetError: 43865:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1389.642454] LNet: Removed LNI 192.168.202.127@tcp [ 1390.623717] Key type .llcrypt unregistered [ 1390.625925] Key type ._llcrypt unregistered [ 1416.886382] Key type ._llcrypt registered [ 1416.892671] Key type .llcrypt registered [ 1417.064978] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 1418.410147] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1418.530092] alg: No test for adler32 (adler32-zlib) [ 1419.669491] Lustre: Lustre: Build Version: 2.17.58_2_g69c2166 [ 1419.975652] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 1421.656970] Key type lgssc registered [ 1422.803858] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1431.684563] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1431.710319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1433.280514] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1439.013324] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 1443.814805] LustreError: 45282:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1443.848173] LustreError: 45282:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1448.821674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1448.933533] LustreError: 45283:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1449.186743] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1455.341298] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 1463.761846] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1471.162661] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1478.331090] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1478.795406] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1485.916723] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 1494.341815] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1503.420364] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1513.956974] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1513.991603] Lustre: Skipped 1 previous similar message [ 1513.994380] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1514.025916] Lustre: Skipped 1 previous similar message [ 1515.501932] Lustre: lustre-OST0000: Not available for connect from 192.168.202.27@tcp (stopping) [ 1517.622668] Lustre: server umount lustre-OST0000 complete [ 1539.554271] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1539.559882] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1539.575912] Lustre: Skipped 1 previous similar message [ 1539.590202] Lustre: Skipped 1 previous similar message [ 1542.927702] Lustre: server umount lustre-MDT0000 complete [ 1544.673994] LustreError: 45279:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1544.683917] LustreError: 45279:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1547.407591] LustreError: 47126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788625914 with bad export cookie 13724063044211257567 [ 1547.413984] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1547.416216] LustreError: 47126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1554.031824] Lustre: server umount lustre-MDT0001 complete [ 1568.102541] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 1571.242247] Key type lgssc unregistered [ 1571.584776] LNet: 49016:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1571.604816] LNetError: 49016:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1571.630503] LNet: Removed LNI 192.168.202.127@tcp [ 1572.617635] Key type .llcrypt unregistered [ 1572.621831] Key type ._llcrypt unregistered [ 1598.826708] Key type ._llcrypt registered [ 1598.828527] Key type .llcrypt registered [ 1599.033774] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 1600.830810] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1600.931214] alg: No test for adler32 (adler32-zlib) [ 1602.038502] Lustre: Lustre: Build Version: 2.17.58_2_g69c2166 [ 1602.247913] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 1603.904424] Key type lgssc registered [ 1605.264721] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1613.737479] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1613.774041] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1615.245467] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1620.197333] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 1625.573692] LustreError: 50431:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1625.607866] LustreError: 50431:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1630.575538] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1630.688939] LustreError: 50430:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1630.982488] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1635.676195] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 1644.716300] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1651.522130] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1659.210257] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1659.473706] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1660.549616] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1666.620532] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 1675.707857] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1682.217705] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1685.498877] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1698.046597] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-0512h34m22s [ 1699.706996] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1702.881589] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1702.888165] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1702.920669] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1704.035668] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1704.049208] Lustre: Skipped 5 previous similar messages [ 1705.473847] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.27@tcp (stopping) [ 1705.478035] Lustre: Skipped 1 previous similar message [ 1708.553593] Lustre: server umount lustre-MDT0000 complete [ 1710.530297] LustreError: 50426:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1712.403687] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1716.678684] LustreError: 52271:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1716.696218] LustreError: 52271:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1720.690947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1720.810961] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1721.031249] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1721.081455] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1722.041447] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1726.718273] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 1730.769674] Lustre: DEBUG MARKER: Wait for df (36690) ... [ 1732.356405] Lustre: DEBUG MARKER: done [ 1736.681180] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1736.689498] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1736.702518] Lustre: Skipped 2 previous similar messages [ 1736.706816] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1737.698070] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1737.704732] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1742.481785] Lustre: server umount lustre-OST0000 complete [ 1747.943217] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1747.954094] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1747.954126] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1747.958943] Lustre: Skipped 1 previous similar message [ 1747.977354] Lustre: Skipped 3 previous similar messages [ 1758.177948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1758.183845] Lustre: Skipped 5 previous similar messages [ 1761.248142] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1761.541874] Lustre: server umount lustre-MDT0000 complete [ 1763.239295] LustreError: 52271:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1763.259389] LustreError: 52271:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 1765.650442] LustreError: 53059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788626132 with bad export cookie 13512886728464195712 [ 1765.658066] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1765.670643] LustreError: 53059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1766.141160] Lustre: server umount lustre-MDT0001 complete [ 1780.239503] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 1782.946059] Key type lgssc unregistered [ 1783.208660] LNet: 55360:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1783.223984] LNetError: 55360:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1783.246432] LNet: Removed LNI 192.168.202.127@tcp [ 1784.056404] Key type .llcrypt unregistered [ 1784.058711] Key type ._llcrypt unregistered [ 1826.908810] Key type ._llcrypt registered [ 1826.911528] Key type .llcrypt registered [ 1827.068659] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 1828.424856] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1828.433811] alg: No test for adler32 (adler32-zlib) [ 1829.622568] Lustre: Lustre: Build Version: 2.17.58_2_g69c2166 [ 1830.075237] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 1831.817429] Key type lgssc registered [ 1833.171771] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1840.772761] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1840.805774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1842.141688] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1842.183688] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1842.352952] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1848.348803] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 1857.535741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1857.672646] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1857.697588] Lustre: 57566:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1857.971976] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1863.038942] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 1871.671452] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1877.868867] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1885.634282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1885.894161] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1886.137301] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1893.096157] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1893.321638] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 1901.737254] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1909.267044] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-0512h37m54s [ 1911.034894] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1919.272846] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1921.968747] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1922.233845] LustreError: 60356:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1942.533433] Lustre: lustre-MDT0000: Client bcfc8ef8-c2bb-4aff-83e9-33c16004db93 (at 192.168.202.27@tcp) reconnecting [ 1962.272559] LustreError: 60356:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1964.861656] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1967.533391] Lustre: DEBUG MARKER: done [ 1974.758171] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1974.764777] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1974.787486] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1975.287739] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1975.291930] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1975.333694] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1975.336475] Lustre: Skipped 1 previous similar message [ 1980.508570] LustreError: 59412:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1980.521334] LustreError: 59412:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1980.596381] Lustre: server umount lustre-OST0000 complete [ 1986.019638] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1986.022569] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1986.042868] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1986.063836] Lustre: Skipped 1 previous similar message [ 1990.904452] Lustre: server umount lustre-MDT0000 complete [ 1992.164980] LustreError: 57577:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1992.192794] LustreError: 57577:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1995.444527] LustreError: 57559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788626362 with bad export cookie 11854804918890254068 [ 1995.455910] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1995.798089] Lustre: server umount lustre-MDT0001 complete [ 2009.779824] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 2013.327676] Key type lgssc unregistered [ 2013.616739] LNet: 61896:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2013.622140] LNetError: 61896:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2013.633618] LNet: Removed LNI 192.168.202.127@tcp [ 2014.450902] Key type .llcrypt unregistered [ 2014.452718] Key type ._llcrypt unregistered [ 2062.166728] Key type ._llcrypt registered [ 2062.170311] Key type .llcrypt registered [ 2062.229509] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 2063.033319] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2063.056200] alg: No test for adler32 (adler32-zlib) [ 2064.141498] Lustre: Lustre: Build Version: 2.17.58_2_g69c2166 [ 2064.462972] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 2066.233940] Key type lgssc registered [ 2067.662181] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2070.972166] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2070.996099] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2072.334964] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2072.377972] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2072.526787] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2073.695062] Lustre: Failing over lustre-MDT0000 [ 2074.141539] Lustre: server umount lustre-MDT0000 complete [ 2082.326561] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 12:40:47 (1788626447) [ 2089.188430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2089.463230] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2093.910490] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 2104.082897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2104.141220] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2104.164638] Lustre: 64696:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2104.373691] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2110.359876] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 2119.098979] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2125.508877] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2133.040059] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2133.236741] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2133.408884] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2139.915204] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2140.881502] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 2149.666780] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2156.050505] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2165.732845] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2165.739742] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2165.754856] Lustre: Skipped 1 previous similar message [ 2169.611383] Lustre: server umount lustre-MDT0000 complete [ 2173.302288] LustreError: 64686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788626540 with bad export cookie 5624859346522383821 [ 2173.316250] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2174.242975] Lustre: server umount lustre-MDT0001 complete [ 2178.552681] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2184.991392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2191.392137] Lustre: 63617:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788626542/real 1788626542] req@000000006530005c x1875510748554624/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788626558 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2191.401653] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2191.406631] Lustre: Skipped 1 previous similar message [ 2194.402194] Lustre: 63618:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788626545/real 1788626545] req@0000000024deff70 x1875510748554752/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788626561 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2194.457505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2197.472472] Lustre: 63619:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788626548/real 1788626548] req@00000000ad994638 x1875510748554880/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788626564 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2198.560824] LustreError: 63616:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000a68f27e2 x1875510748555648/t0(0) o250->MGC192.168.202.127@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2198.919396] LustreError: 66543:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2198.929333] LustreError: 66543:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2198.981327] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2199.084034] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2199.971809] Lustre: 63620:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788626550/real 1788626550] req@000000006e2ceca6 x1875510748555136/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788626566 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2201.004657] LustreError: 66544:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2202.091219] LustreError: 68467:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2202.124067] LustreError: 68467:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2204.131108] Lustre: 63618:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788626555/real 1788626555] req@00000000b492900b x1875510748555264/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788626571 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2205.085196] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 2207.216897] LustreError: 66543:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2207.256550] LustreError: 66543:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2212.340283] LustreError: 68467:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2212.355522] LustreError: 68467:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2215.161946] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2215.653050] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2220.807954] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2222.410677] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 2232.268475] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2242.041262] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2249.728939] Lustre: server umount lustre-MDT0000 complete [ 2251.745414] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2251.757491] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2251.802492] LustreError: 68465:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2251.833209] LustreError: 68465:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2254.562504] LustreError: 64687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788626621 with bad export cookie 5624859346522386096 [ 2254.566283] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2254.572273] LustreError: 64687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2255.089406] Lustre: server umount lustre-MDT0001 complete [ 2262.532806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2272.161211] Lustre: 63619:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788626623/real 1788626623] req@00000000ff570ab4 x1875510748583808/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788626639 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2272.192428] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2272.214543] Lustre: Skipped 2 previous similar messages [ 2273.155156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2280.427077] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x4e0f83c61d00b413 [ 2280.447605] Lustre: MGC192.168.202.127@tcp: Connection restored to 0@lo (at 0@lo) [ 2280.463017] Lustre: Skipped 1 previous similar message [ 2281.172248] LustreError: 66543:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2281.357740] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2281.471476] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2287.648790] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 2291.754589] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2297.069339] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2302.884907] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 2302.951269] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2311.863625] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2318.183622] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2328.548610] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2328.567565] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2328.571499] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2328.585368] Lustre: Skipped 1 previous similar message [ 2328.591170] Lustre: Skipped 2 previous similar messages [ 2331.862253] Lustre: server umount lustre-MDT0000 complete [ 2333.668916] LustreError: 71150:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2333.696303] LustreError: 71150:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 2336.299437] LustreError: 64687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788626703 with bad export cookie 5624859346522387475 [ 2336.309301] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2336.320387] LustreError: 64687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2336.861031] Lustre: server umount lustre-MDT0001 complete [ 2342.947850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2347.033764] Lustre: DEBUG MARKER: files compared the same [ 2359.864630] Lustre: server umount lustre-OST0000 complete [ 2381.058890] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 2384.175460] Key type lgssc unregistered [ 2384.518062] LNet: 74426:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2384.523892] LNetError: 74426:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2384.546977] LNet: Removed LNI 192.168.202.127@tcp [ 2385.667365] Key type .llcrypt unregistered [ 2385.669836] Key type ._llcrypt unregistered [ 2410.416346] Key type ._llcrypt registered [ 2410.418113] Key type .llcrypt registered [ 2410.535943] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 2411.620048] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2411.692738] alg: No test for adler32 (adler32-zlib) [ 2412.869817] Lustre: Lustre: Build Version: 2.17.58_2_g69c2166 [ 2413.114083] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 2414.832199] Key type lgssc registered [ 2416.177046] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2424.274645] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2424.298866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2425.920278] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2430.763521] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug +malloc all [ 2435.046568] LustreError: 75840:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2435.082234] LustreError: 75840:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2439.706123] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2440.014348] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2444.726708] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug +malloc all [ 2452.396571] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2459.169332] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2466.418697] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2466.768716] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2470.902487] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2472.680584] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug +malloc all [ 2480.127929] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2491.363401] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2491.378391] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2491.397951] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2494.435989] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2494.438090] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2494.439283] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2496.380185] Lustre: server umount lustre-OST0000 complete [ 2501.604382] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2501.613329] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2501.626699] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2501.632521] Lustre: Skipped 1 previous similar message [ 2504.673584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2504.681557] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2507.013616] Lustre: server umount lustre-MDT0000 complete [ 2509.792941] LustreError: 75835:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2509.807295] LustreError: 75835:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2511.064815] LustreError: 77685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788626878 with bad export cookie 58965893557972199 [ 2511.067366] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2511.084783] LustreError: 77685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2511.454707] Lustre: server umount lustre-MDT0001 complete [ 2522.034257] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 2524.592322] Key type lgssc unregistered [ 2524.849654] LNet: 79373:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2524.861691] LNetError: 79373:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2524.869473] LNet: Removed LNI 192.168.202.127@tcp [ 2525.621811] Key type .llcrypt unregistered [ 2525.623916] Key type ._llcrypt unregistered [ 2547.930292] Key type ._llcrypt registered [ 2547.932673] Key type .llcrypt registered [ 2548.060475] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 2549.264941] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2549.509843] alg: No test for adler32 (adler32-zlib) [ 2550.576755] Lustre: Lustre: Build Version: 2.17.58_2_g69c2166 [ 2550.855092] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 2552.552175] Key type lgssc registered [ 2553.610374] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2561.007331] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2561.024170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2576.672421] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2581.792414] LustreError: 80760:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.127@tcp: failed processing log, type 4: rc = -110 [ 2608.416215] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2614.805972] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 2624.303630] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2633.530714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2633.678358] Lustre: *** cfs_fail_loc=706, val=0*** [ 2634.202826] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2634.270462] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2639.275886] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 2639.337362] LustreError: 80786:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2639.361984] LustreError: 80786:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2643.437840] LustreError: 81845:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2643.454959] LustreError: 81845:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2648.490487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2648.549824] LustreError: 80786:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2648.572648] LustreError: 80786:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2648.929176] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2653.678838] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 2661.213734] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2666.777928] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2669.545279] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2669.552111] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2669.559789] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2669.578878] Lustre: Skipped 1 previous similar message [ 2674.665211] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2674.691817] Lustre: Skipped 2 previous similar messages [ 2679.782932] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2679.798762] Lustre: Skipped 1 previous similar message [ 2683.360168] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2683.510295] Lustre: server umount lustre-OST0000 complete [ 2690.024594] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2690.025505] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2690.043665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2693.876764] Lustre: server umount lustre-MDT0000 complete [ 2695.143457] LustreError: 82298:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2695.174014] LustreError: 82298:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2698.161317] LustreError: 80767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788627065 with bad export cookie 2232052292370728568 [ 2698.163381] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2698.170427] LustreError: 80767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2698.551887] Lustre: server umount lustre-MDT0001 complete [ 2712.572495] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 2715.526186] Key type lgssc unregistered [ 2715.979635] LNet: 84378:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2715.994701] LNetError: 84378:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2717.031365] LNet: Removed LNI 192.168.202.127@tcp [ 2718.017617] Key type .llcrypt unregistered [ 2718.020331] Key type ._llcrypt unregistered [ 2746.181411] Key type ._llcrypt registered [ 2746.184885] Key type .llcrypt registered [ 2746.302349] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 2747.598553] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2747.615366] alg: No test for adler32 (adler32-zlib) [ 2748.798935] Lustre: Lustre: Build Version: 2.17.58_2_g69c2166 [ 2749.089558] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 2750.777039] Key type lgssc registered [ 2752.491371] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2761.440045] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2761.490891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2767.395024] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 2775.687768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2776.335189] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2781.665636] LustreError: 86483:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2781.696242] LustreError: 86483:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2782.347772] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 2786.788016] LustreError: 86486:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2786.806917] LustreError: 86486:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2791.790437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2791.910770] LustreError: 86487:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2791.939972] LustreError: 86487:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2792.387799] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2797.539708] LustreError: 86486:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2797.578653] LustreError: 86486:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 2800.592429] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 2801.636653] LustreError: 86487:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2801.667858] LustreError: 86487:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2811.116918] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2811.128024] Lustre: MGS: Client 25c2bacf-20a8-4f3b-8c53-36959ecdde89 (at 0@lo) reconnecting [ 2811.143894] LustreError: 86497:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2811.294347] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2811.367214] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2818.365333] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 2836.966109] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2836.971746] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2836.995888] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2837.014179] Lustre: Skipped 1 previous similar message [ 2837.038633] Lustre: Skipped 1 previous similar message [ 2840.086561] Lustre: server umount lustre-OST0000 complete [ 2847.201761] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2847.206046] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2847.219197] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2847.251105] Lustre: Skipped 1 previous similar message [ 2851.387974] Lustre: server umount lustre-MDT0000 complete [ 2852.326859] LustreError: 86486:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2852.337209] LustreError: 86486:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2856.746861] Lustre: server umount lustre-MDT0001 complete [ 2861.348762] Lustre: server umount MGS complete [ 2874.583392] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 12:53:59 (1788627239) [ 2906.037979] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_hostid [ 2915.836815] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 2956.854345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2961.155711] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 2971.919782] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2972.307508] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2972.425577] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2972.542097] Lustre: lustre-MDT0001: new disk, initializing [ 2972.662131] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2972.713450] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2972.722743] Lustre: 93669:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2972.735344] LustreError: 93669:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2977.579777] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 2988.886488] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2989.122330] Lustre: lustre-OST0000: new disk, initializing [ 2989.125810] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2989.132152] Lustre: 94434:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2989.266378] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2995.800503] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 2996.837460] Lustre: 94787:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000b484148f x1875511466829440/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788627379 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3005.142553] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3011.807911] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3011.814468] Lustre: MGS: Client 0084ebac-d37b-4d75-8ff3-98f80f232778 (at 0@lo) reconnecting [ 3011.860026] Lustre: 92937:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3011.875484] LustreError: 93655:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 3011.885238] LustreError: 93655:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 3011.905665] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3011.969634] Lustre: lustre-MDT0000: new disk, initializing [ 3012.073504] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3012.100536] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3012.125926] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3013.159392] Lustre: 95503:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000004fb5104f x1875511466836480/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788627396 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3016.946853] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 3018.343169] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3022.263963] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3022.273689] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3022.399323] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 3048.420608] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3048.426172] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3048.438282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3048.444379] Lustre: Skipped 1 previous similar message [ 3051.504967] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3051.513094] Lustre: Skipped 2 previous similar messages [ 3053.693538] Lustre: server umount lustre-OST0000 complete [ 3061.729852] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3061.735992] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3061.749019] LustreError: Skipped 1 previous similar message [ 3061.752984] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3061.778627] Lustre: Skipped 2 previous similar messages [ 3061.801763] Lustre: Skipped 1 previous similar message [ 3063.984199] Lustre: server umount lustre-MDT0000 complete [ 3066.867799] LustreError: 93644:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3066.907253] LustreError: 93644:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3074.810071] Lustre: server umount lustre-MDT0001 complete [ 3079.685788] Lustre: server umount MGS complete [ 3093.405572] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 12:57:38 (1788627458) [ 3116.197931] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 3119.480733] Key type lgssc unregistered [ 3119.827622] LNet: 98002:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3119.833197] LNetError: 98002:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3119.845320] LNet: Removed LNI 192.168.202.127@tcp [ 3120.956400] Key type .llcrypt unregistered [ 3120.961046] Key type ._llcrypt unregistered [ 3138.157436] Key type ._llcrypt registered [ 3138.160613] Key type .llcrypt registered [ 3138.279163] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 3139.411094] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3139.449703] alg: No test for adler32 (adler32-zlib) [ 3140.494566] Lustre: Lustre: Build Version: 2.17.58_2_g69c2166 [ 3140.734529] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 3142.546694] Key type lgssc registered [ 3143.730567] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3152.686397] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3152.714989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3154.199531] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3159.625590] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 3164.645689] LustreError: 99221:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3164.676900] LustreError: 99221:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3168.594649] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3168.957150] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3174.469930] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 3184.819175] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3192.567889] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3201.827296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3202.303388] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3209.464444] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 3218.933643] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3224.878572] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3234.274621] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3234.282666] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3234.307267] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3235.297745] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3235.305967] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3235.327138] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3235.346163] Lustre: Skipped 1 previous similar message [ 3239.892940] Lustre: server umount lustre-OST0000 complete [ 3245.546957] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3245.557960] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3245.580752] Lustre: Skipped 1 previous similar message [ 3245.596749] Lustre: Skipped 1 previous similar message [ 3250.668658] LustreError: 99961:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3250.741147] Lustre: server umount lustre-MDT0000 complete [ 3254.957108] LustreError: 102027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788627621 with bad export cookie 17200513186156136326 [ 3254.970765] LustreError: 102027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3254.973115] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3255.379312] Lustre: server umount lustre-MDT0001 complete [ 3267.213640] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 3270.942894] Key type lgssc unregistered [ 3271.229589] LNet: 102777:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3271.242266] LNetError: 102777:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3271.258108] LNet: Removed LNI 192.168.202.127@tcp [ 3271.917780] Key type .llcrypt unregistered [ 3271.918886] Key type ._llcrypt unregistered [ 3291.138770] Key type ._llcrypt registered [ 3291.144719] Key type .llcrypt registered [ 3291.263610] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 3292.454087] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3292.488100] alg: No test for adler32 (adler32-zlib) [ 3293.890646] Lustre: Lustre: Build Version: 2.17.58_2_g69c2166 [ 3294.159570] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 3295.857733] Key type lgssc registered [ 3297.619715] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3305.888652] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3305.907893] LustreError: 103858:0:(tgt_mount.c:2573:server_fill_super()) cfs_race id 716 sleeping [ 3305.957432] LustreError: 103870:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 waking [ 3305.963844] LustreError: 103858:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 awake: rc=4952 [ 3305.968865] LustreError: 103858:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3305.981879] LustreError: 103858:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3305.984850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3307.632662] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3317.081832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3317.464728] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3323.104275] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 3330.479875] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing lsmod [ 3334.554152] LustreError: 105076:0:(tgt_mount.c:2573:server_fill_super()) cfs_race id 716 sleeping [ 3334.570264] LustreError: 105077:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 waking [ 3334.574658] LustreError: 105076:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 awake: rc=4992 [ 3334.580137] LustreError: 105076:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3334.590806] LustreError: 105076:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3334.596580] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3335.037692] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3340.261982] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3340.271860] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3340.303112] Lustre: Skipped 1 previous similar message [ 3342.373432] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3342.390369] Lustre: Skipped 4 previous similar messages [ 3345.268378] Lustre: server umount lustre-MDT0000 complete [ 3347.437462] LustreError: 103902:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3347.463342] LustreError: 103902:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3350.498204] LustreError: 103880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788627717 with bad export cookie 3381757539985107723 [ 3350.505400] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3350.515778] LustreError: 103880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3350.627860] LustreError: 104339:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000000414d776 x1875512038282752/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3350.942961] Lustre: server umount lustre-MDT0001 complete [ 3365.371949] Lustre: server umount lustre-OST0000 complete [ 3376.420673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3377.030029] LustreError: 106144:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3377.175901] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3382.252843] LustreError: 106145:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3382.589891] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 3387.372928] LustreError: 106144:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3392.234064] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3392.482490] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3392.696650] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3397.685718] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 3406.560225] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3412.790163] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3420.471643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3420.763638] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3422.848072] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3427.797778] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 3435.573267] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3451.278761] Lustre: server umount lustre-OST0000 complete [ 3458.019512] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3458.026050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3458.036237] Lustre: Skipped 2 previous similar messages [ 3458.049320] Lustre: Skipped 1 previous similar message [ 3463.153228] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3463.167475] Lustre: Skipped 1 previous similar message [ 3465.331341] Lustre: server umount lustre-MDT0000 complete [ 3468.257171] Lustre: 103259:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788627819/real 1788627819] req@000000003fd77d61 x1875512038313344/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788627835 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3468.263109] LustreError: 106145:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3468.313498] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3468.364580] LustreError: 106145:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3469.280828] Lustre: 103258:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788627819/real 1788627819] req@00000000f21e6f65 x1875512038313728/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788627835 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3469.919748] LustreError: 106124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788627836 with bad export cookie 3381757539985108423 [ 3469.921614] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3469.934422] LustreError: 106124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3470.398823] Lustre: server umount lustre-MDT0001 complete [ 3484.008815] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 3487.608869] Key type lgssc unregistered [ 3488.213630] LNet: 109697:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3488.223176] LNetError: 109697:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3488.238760] LNet: Removed LNI 192.168.202.127@tcp [ 3489.356630] Key type .llcrypt unregistered [ 3489.358324] Key type ._llcrypt unregistered [ 3516.756276] Key type ._llcrypt registered [ 3516.767439] Key type .llcrypt registered [ 3516.933611] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 3518.443442] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3518.574070] alg: No test for adler32 (adler32-zlib) [ 3519.883869] Lustre: Lustre: Build Version: 2.17.58_2_g69c2166 [ 3520.133235] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 3521.912501] Key type lgssc registered [ 3523.399465] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3533.882717] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3533.894304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3535.800981] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3540.911452] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 3545.064271] LustreError: 111110:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3545.084762] LustreError: 111110:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3550.185155] LustreError: 111111:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3552.200881] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3552.456374] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3557.032780] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 3568.597630] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3576.604932] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3586.203846] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3586.651220] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3594.789175] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 3605.078537] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3605.891727] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3612.363679] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3616.427254] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3623.394539] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3623.406153] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3623.430434] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3625.448382] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3625.452721] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3625.483303] Lustre: Skipped 1 previous similar message [ 3627.539202] Lustre: server umount lustre-OST0000 complete [ 3634.662975] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3634.681427] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3634.725957] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3640.805490] LustreError: 113607:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3640.837328] LustreError: 113607:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3641.427694] Lustre: server umount lustre-MDT0000 complete [ 3645.923975] LustreError: 111110:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3645.953368] LustreError: 111110:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3646.639529] LustreError: 111090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788628013 with bad export cookie 16352094636508102328 [ 3646.648550] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3646.657384] LustreError: 111090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3647.173091] Lustre: server umount lustre-MDT0001 complete [ 3662.780279] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 3666.316767] Key type lgssc unregistered [ 3666.813911] LNet: 114753:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3666.828116] LNetError: 114753:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3666.847600] LNet: Removed LNI 192.168.202.127@tcp [ 3668.076469] Key type .llcrypt unregistered [ 3668.078689] Key type ._llcrypt unregistered [ 3690.221853] Key type ._llcrypt registered [ 3690.225812] Key type .llcrypt registered [ 3690.351352] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 3691.573259] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3691.670708] alg: No test for adler32 (adler32-zlib) [ 3692.935422] Lustre: Lustre: Build Version: 2.17.58_2_g69c2166 [ 3693.356896] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 3695.130353] Key type lgssc registered [ 3696.764172] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3706.688276] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3706.732758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3708.200765] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3714.294958] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 3718.628106] LustreError: 115974:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3718.656130] LustreError: 115974:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3723.745798] LustreError: 115973:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3724.269669] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3724.583948] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3729.653425] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 3739.750820] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3745.970264] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3753.615731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3753.866277] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3754.878471] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3760.787974] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 3769.717832] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3780.581096] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3780.586650] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3780.627147] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3782.936357] Lustre: server umount lustre-OST0000 complete [ 3789.811271] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3789.831888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3789.832475] Lustre: Skipped 1 previous similar message [ 3789.842460] Lustre: Skipped 1 previous similar message [ 3794.398188] Lustre: server umount lustre-MDT0000 complete [ 3794.914127] LustreError: 115969:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3794.928398] LustreError: 115969:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3798.832159] LustreError: 115954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788628165 with bad export cookie 14329545772273892123 [ 3798.834148] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3798.839109] LustreError: 115954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3808.538760] Lustre: server umount lustre-MDT0001 complete [ 3820.914567] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 3824.702908] Key type lgssc unregistered [ 3825.209799] LNet: 119510:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3825.223601] LNetError: 119510:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3825.265652] LNet: Removed LNI 192.168.202.127@tcp [ 3826.400505] Key type .llcrypt unregistered [ 3826.402291] Key type ._llcrypt unregistered [ 3854.354906] Key type ._llcrypt registered [ 3854.364884] Key type .llcrypt registered [ 3854.510196] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 3855.997852] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3856.096470] alg: No test for adler32 (adler32-zlib) [ 3857.255945] Lustre: Lustre: Build Version: 2.17.58_2_g69c2166 [ 3857.567504] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 3859.248269] Key type lgssc registered [ 3860.902753] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3870.313791] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3870.330678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3871.845724] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3876.856444] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 3881.454611] LustreError: 120923:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3881.487931] LustreError: 120923:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3886.562565] LustreError: 120924:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3888.369101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3888.951913] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3895.575750] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 3906.666920] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3915.269100] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3925.837286] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3926.279438] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3934.744875] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 3941.689752] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3943.031915] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3951.528396] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3955.915923] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3955.927891] Lustre: Skipped 2 previous similar messages [ 3959.881500] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3959.919600] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3961.354364] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3961.359072] Lustre: Skipped 2 previous similar messages [ 3970.182555] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3970.200033] Lustre: Skipped 1 previous similar message [ 3976.784755] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.27@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3976.794934] Lustre: Skipped 2 previous similar messages [ 3983.035501] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.27@tcp 192.168.0.[2,10]@tcp [ 3991.641285] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.27/24@tcp in log lustre-MDT0000 [ 3991.646854] Lustre: Skipped 2 previous similar messages [ 3999.895261] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.27/24@tcp [ 3999.908617] Lustre: Skipped 1 previous similar message [ 4007.521520] Lustre: server umount lustre-OST0000 complete [ 4013.537936] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4013.545187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4013.549500] Lustre: Skipped 1 previous similar message [ 4018.659180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4018.667880] Lustre: Skipped 1 previous similar message [ 4022.917571] Lustre: server umount lustre-MDT0000 complete [ 4023.783824] LustreError: 120923:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4023.818756] LustreError: 120923:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4024.805425] Lustre: 120233:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788628375/real 1788628375] req@0000000017b5637e x1875512628684160/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788628391 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4024.836641] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4024.858518] Lustre: Skipped 1 previous similar message [ 4026.883809] LustreError: 120906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788628393 with bad export cookie 2971542012374287328 [ 4026.891061] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4026.903903] LustreError: 120906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4027.277802] Lustre: server umount lustre-MDT0001 complete [ 4038.843326] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 4041.144454] Key type lgssc unregistered [ 4041.475712] LNet: 126019:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4041.502346] LNetError: 126019:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4042.540059] LNet: Removed LNI 192.168.202.127@tcp [ 4043.620319] Key type .llcrypt unregistered [ 4043.625487] Key type ._llcrypt unregistered [ 4076.736666] Key type ._llcrypt registered [ 4076.741362] Key type .llcrypt registered [ 4076.851510] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 4078.295192] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4078.448664] alg: No test for adler32 (adler32-zlib) [ 4079.706414] Lustre: Lustre: Build Version: 2.17.58_2_g69c2166 [ 4080.079549] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 4081.760195] Key type lgssc registered [ 4083.060150] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4091.579665] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4091.602251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4093.149911] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 4093.175939] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4093.196116] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.27/24@tcp [ 4098.945466] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 4103.652315] LustreError: 127577:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4103.683486] LustreError: 127577:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4108.629773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4108.769956] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4108.866071] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4108.890286] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.27/24@tcp [ 4108.923697] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4114.104261] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 4122.118291] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4127.816974] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4136.006311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4136.498935] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4138.559036] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 4143.859496] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 4150.876247] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4165.102927] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4165.114576] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4165.136624] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4169.185671] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4169.193165] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4169.194476] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4170.976714] Lustre: server umount lustre-OST0000 complete [ 4175.330152] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4175.334845] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4175.343762] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4175.349679] Lustre: Skipped 1 previous similar message [ 4179.425933] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4179.430063] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4181.673645] Lustre: server umount lustre-MDT0000 complete [ 4184.546541] LustreError: 127578:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4184.566552] LustreError: 127578:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4185.967642] LustreError: 127560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788628552 with bad export cookie 12779762466235636476 [ 4185.979696] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4185.987861] LustreError: 127560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4186.404361] Lustre: server umount lustre-MDT0001 complete [ 4198.832617] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 4202.136410] Key type lgssc unregistered [ 4202.588147] LNet: 131164:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4202.596426] LNetError: 131164:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4202.618761] LNet: Removed LNI 192.168.202.127@tcp [ 4203.365157] Key type .llcrypt unregistered [ 4203.367334] Key type ._llcrypt unregistered [ 4234.041856] Key type ._llcrypt registered [ 4234.044483] Key type .llcrypt registered [ 4234.159964] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_hostid [ 4249.975741] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 4251.372546] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4251.395320] alg: No test for adler32 (adler32-zlib) [ 4252.845208] Lustre: Lustre: Build Version: 2.17.58_2_g69c2166 [ 4253.129048] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 4254.888231] Key type lgssc registered [ 4256.319487] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4294.728662] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4294.759644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4296.167816] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4296.222483] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4296.341934] Lustre: lustre-MDT0000: new disk, initializing [ 4296.439173] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4296.465417] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4302.668973] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 4314.838769] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4315.033082] Lustre: 134562:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4315.113280] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4315.119020] Lustre: Skipped 1 previous similar message [ 4315.186230] Lustre: lustre-MDT0001: new disk, initializing [ 4315.298578] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4315.354950] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4315.372947] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4320.689397] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 4332.353964] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4338.644940] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4345.674479] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4345.934967] Lustre: lustre-OST0000: new disk, initializing [ 4345.939543] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4345.944747] Lustre: 136531:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4346.061234] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4353.241436] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 4355.693207] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4355.709311] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4355.844965] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4363.361033] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4368.298438] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4368.598307] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4374.459673] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4374.827487] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4381.678693] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4381.679039] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4381.682992] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4381.696460] LustreError: Skipped 1 previous similar message [ 4381.712707] Lustre: Skipped 1 previous similar message [ 4386.804422] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4386.813695] Lustre: Skipped 1 previous similar message [ 4391.392222] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4391.678535] Lustre: server umount lustre-OST0000 complete [ 4397.028142] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4397.034374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4397.040651] Lustre: Skipped 1 previous similar message [ 4397.049266] Lustre: Skipped 2 previous similar messages [ 4401.860626] Lustre: server umount lustre-MDT0000 complete [ 4405.891756] LustreError: 134552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788628772 with bad export cookie 1159212817287654978 [ 4405.897443] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4405.899118] LustreError: 134552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4406.235122] Lustre: server umount lustre-MDT0001 complete [ 4415.967898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4416.439789] LustreError: 138748:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4416.448084] LustreError: 138748:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4416.524608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4421.611688] LustreError: 138749:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4422.318617] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 4426.730129] LustreError: 138748:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4431.848095] LustreError: 138749:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4432.340133] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4432.854934] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4438.165672] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 4446.613566] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4453.650697] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4460.707347] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4460.983357] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4467.271500] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 4475.955287] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4482.060512] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4482.345417] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4488.325863] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4488.698257] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4500.233940] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4500.406699] Lustre: lustre-OST0001: new disk, initializing [ 4500.410977] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4500.418417] Lustre: 142121:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4500.536314] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4507.275482] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4507.285621] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4507.404959] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4508.429691] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 4517.974514] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4518.319986] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4523.342540] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4523.589913] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4530.996851] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9832c5d9a000.ost_server_uuid 50 [ 4532.871258] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9832c5d9a000.ost_server_uuid in IDLE state after 0 sec [ 4543.468986] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4543.475679] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4543.485649] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4543.496836] Lustre: Skipped 1 previous similar message [ 4543.519764] LustreError: Skipped 1 previous similar message [ 4545.402418] Lustre: server umount lustre-OST0001 complete [ 4548.585610] LustreError: 140571:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4548.608937] LustreError: 140571:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4553.699164] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4553.710166] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4553.720822] Lustre: Skipped 1 previous similar message [ 4553.725369] Lustre: Skipped 1 previous similar message [ 4555.985789] Lustre: server umount lustre-OST0000 complete [ 4560.895264] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4560.913506] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4563.941691] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4563.958094] Lustre: Skipped 3 previous similar messages [ 4566.697865] Lustre: server umount lustre-MDT0000 complete [ 4569.062549] LustreError: 139714:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4569.075431] LustreError: 139714:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4570.834383] LustreError: 138730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788628937 with bad export cookie 1159212817287655860 [ 4570.838696] LustreError: 138730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4570.842960] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4571.268916] Lustre: server umount lustre-MDT0001 complete [ 4593.178084] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 4596.764409] Key type lgssc unregistered [ 4597.076695] LNet: 145123:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4597.085789] LNetError: 145123:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4597.107622] LNet: Removed LNI 192.168.202.127@tcp [ 4598.026866] Key type .llcrypt unregistered [ 4598.032256] Key type ._llcrypt unregistered [ 4644.379798] Key type ._llcrypt registered [ 4644.382789] Key type .llcrypt registered [ 4644.489544] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_hostid [ 4663.079508] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 4663.961905] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4664.051530] alg: No test for adler32 (adler32-zlib) [ 4665.039851] Lustre: Lustre: Build Version: 2.17.58_2_g69c2166 [ 4665.225169] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 4666.888151] Key type lgssc registered [ 4668.022198] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4706.064623] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4706.141301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4707.435889] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4707.465218] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4707.558557] Lustre: lustre-MDT0000: new disk, initializing [ 4707.674520] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4707.690799] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4712.577972] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 4723.671802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4723.813887] Lustre: 149278:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4723.856866] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4723.859769] Lustre: Skipped 1 previous similar message [ 4723.912908] Lustre: lustre-MDT0001: new disk, initializing [ 4723.998870] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4724.025117] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4724.031233] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4728.236074] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 4738.929861] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4745.568683] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4754.161684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4754.609534] Lustre: lustre-OST0000: new disk, initializing [ 4754.613615] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4754.621489] Lustre: 151245:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4754.720981] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4762.222342] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 4762.691976] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4762.707777] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4762.821936] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4773.076348] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4778.201672] Lustre: Failing over lustre-OST0000 [ 4778.469498] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4778.483508] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4778.484016] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4778.484436] Lustre: Skipped 1 previous similar message [ 4778.506424] Lustre: Skipped 1 previous similar message [ 4780.718621] Lustre: server umount lustre-OST0000 complete [ 4799.374832] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4799.638917] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4799.646688] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4807.430728] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 4807.624600] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4813.023529] Lustre: Failing over lustre-MDT0000 [ 4813.338850] Lustre: server umount lustre-MDT0000 complete [ 4814.312862] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4814.344377] Lustre: Skipped 1 previous similar message [ 4822.985163] LustreError: 149285:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4823.006403] LustreError: 149285:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 4823.524588] LustreError: 149286:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4823.541726] LustreError: 149286:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4828.103286] LustreError: 149284:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4830.688325] Lustre: 147503:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788629181/real 1788629181] req@000000009252259b x1875513475905152/t0(0) o400->MGC192.168.202.127@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788629197 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4830.729896] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4830.752210] LustreError: 151493:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4830.777057] LustreError: 151493:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 4835.809684] LustreError: 149284:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4835.846330] LustreError: 149284:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4836.207077] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4840.933180] LustreError: 147500:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000006be202b6 x1875513475910016/t0(0) o250->MGC192.168.202.127@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4841.268412] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 4841.376630] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4841.444033] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4843.478666] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4846.580675] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4846.608184] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 4846.710105] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4847.125175] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 4856.804075] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4856.809425] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4856.818809] Lustre: Skipped 1 previous similar message [ 4856.849081] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4856.861417] Lustre: Skipped 1 previous similar message [ 4862.046678] Lustre: server umount lustre-OST0000 complete [ 4867.050737] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4867.067976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4867.075280] Lustre: Skipped 1 previous similar message [ 4872.166159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4872.188966] Lustre: Skipped 1 previous similar message [ 4876.259423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4876.266612] Lustre: Skipped 1 previous similar message [ 4881.376571] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4881.563570] Lustre: server umount lustre-MDT0000 complete [ 4885.972671] LustreError: 149268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788629252 with bad export cookie 9186857247527610418 [ 4885.974665] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4885.984655] LustreError: 149268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4886.464334] Lustre: server umount lustre-MDT0001 complete [ 4900.401124] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 4903.470342] Key type lgssc unregistered [ 4903.721982] LNet: 154906:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4903.730788] LNetError: 154906:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4903.743238] LNet: Removed LNI 192.168.202.127@tcp [ 4904.684783] Key type .llcrypt unregistered [ 4904.686792] Key type ._llcrypt unregistered [ 4946.046780] Key type ._llcrypt registered [ 4946.048647] Key type .llcrypt registered [ 4946.142314] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 4947.631077] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4947.756719] alg: No test for adler32 (adler32-zlib) [ 4948.877788] Lustre: Lustre: Build Version: 2.17.58_2_g69c2166 [ 4949.129676] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 4950.776182] Key type lgssc registered [ 4952.041799] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4960.132623] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4960.148456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4961.728611] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4967.329292] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 4972.005986] LustreError: 156441:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4972.024168] LustreError: 156441:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4976.043564] LustreError: 156440:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4978.088373] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4981.219526] LustreError: 156441:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4982.267141] LustreError: 156469:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4982.302622] LustreError: 156469:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4982.315744] LustreError: 156469:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 21, retries 0, failed: rc = -5 [ 4982.743827] Lustre: server umount lustre-MDT0000 complete [ 4999.099140] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_hostid [ 5009.880436] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 5052.548300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5052.891710] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5052.923747] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5053.027529] Lustre: lustre-MDT0000: new disk, initializing [ 5053.124476] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5053.153649] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5058.695197] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 5070.174155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5070.340735] Lustre: 159772:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5070.425459] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5070.429968] Lustre: Skipped 1 previous similar message [ 5070.525624] Lustre: lustre-MDT0001: new disk, initializing [ 5070.638519] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5070.697490] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5070.709213] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5075.788883] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 5086.201860] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5094.261340] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5102.730657] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5103.023774] Lustre: lustre-OST0000: new disk, initializing [ 5103.025680] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5103.047758] Lustre: 161742:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5103.123836] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5110.595704] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 5113.000580] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5113.009142] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5113.162682] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5121.129784] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5125.738964] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 5127.604810] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 13:31:33 (1788629493) [ 5394.912645] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5394.915766] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5394.921661] Lustre: Skipped 1 previous similar message [ 5394.939058] Lustre: Skipped 2 previous similar messages [ 5399.923358] Lustre: server umount lustre-MDT0000 complete [ 5400.034906] LustreError: 159780:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5400.063698] LustreError: 159780:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5403.572612] LustreError: 159766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788629770 with bad export cookie 9772264445155149822 [ 5403.582304] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5403.603222] LustreError: 159766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5405.153405] LustreError: 159778:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5405.154228] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5405.174889] Lustre: Skipped 1 previous similar message [ 5405.180909] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5408.160850] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5410.402973] Lustre: server umount lustre-MDT0001 complete [ 5421.551663] Lustre: server umount lustre-OST0000 complete [ 5432.296522] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_hostid [ 5441.880882] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 5483.093656] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5483.439345] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5483.495940] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5483.689827] Lustre: lustre-MDT0000: new disk, initializing [ 5483.842311] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5483.864358] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5489.755353] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 5502.101577] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5502.258526] Lustre: 166418:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5502.337097] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5502.347082] Lustre: Skipped 1 previous similar message [ 5502.446948] Lustre: lustre-MDT0001: new disk, initializing [ 5502.568386] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5502.668124] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5502.682371] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5507.287595] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 5518.389665] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5525.978911] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5534.217788] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5534.787726] Lustre: lustre-OST0000: new disk, initializing [ 5534.794053] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5534.808609] Lustre: 168386:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5534.942982] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5542.500317] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 5542.501372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5542.526712] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5542.709423] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5554.052784] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5559.423964] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5559.692729] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5565.502082] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5565.866957] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5568.485142] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5568.485887] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5568.528036] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5568.540501] Lustre: Skipped 1 previous similar message [ 5573.603221] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5573.633039] Lustre: Skipped 1 previous similar message [ 5574.578222] Lustre: server umount lustre-OST0000 complete [ 5579.238490] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5579.247218] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5579.258267] Lustre: Skipped 1 previous similar message [ 5583.841965] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5583.845959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5583.864478] Lustre: Skipped 2 previous similar messages [ 5585.540179] Lustre: server umount lustre-MDT0000 complete [ 5588.963313] LustreError: 167707:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5588.989749] LustreError: 167707:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5589.541964] LustreError: 166410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788629956 with bad export cookie 9772264445155308652 [ 5589.546258] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5589.554117] LustreError: 166410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5589.866927] Lustre: server umount lustre-MDT0001 complete [ 5600.031528] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 13:39:25 (1788629965) [ 5618.617228] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_hostid [ 5630.173503] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 5673.392993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5673.730325] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5673.809795] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5674.075991] Lustre: lustre-MDT0000: new disk, initializing [ 5674.277520] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5674.325543] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5679.591684] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 5693.866825] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5694.025333] Lustre: 173142:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5694.403664] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5699.347401] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 5710.429171] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5716.731643] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5723.862467] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5724.137579] Lustre: lustre-OST0000: new disk, initializing [ 5724.140294] Lustre: Skipped 1 previous similar message [ 5724.143864] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5724.152992] Lustre: Skipped 2 previous similar messages [ 5724.159142] Lustre: 175121:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5730.591655] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 5733.516517] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5733.531890] Lustre: Skipped 1 previous similar message [ 5733.536885] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5733.633780] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5741.319782] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5749.218198] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5749.221160] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5749.227569] LustreError: Skipped 1 previous similar message [ 5749.241386] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5749.247482] Lustre: Skipped 1 previous similar message [ 5749.266455] Lustre: Skipped 1 previous similar message [ 5754.353430] LustreError: 175588:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5754.377083] LustreError: 175588:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5754.676093] Lustre: server umount lustre-OST0000 complete [ 5761.001175] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5766.761985] Lustre: server umount lustre-MDT0000 complete [ 5771.304328] LustreError: 173135:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788630138 with bad export cookie 9772264445155309534 [ 5771.312801] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5771.324246] LustreError: 173135:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5771.788406] Lustre: server umount lustre-MDT0001 complete [ 5782.818753] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 13:42:27 (1788630147) [ 5803.402411] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_hostid [ 5814.221637] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 5860.050498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5860.486396] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5860.512741] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5860.569257] Lustre: lustre-MDT0000: new disk, initializing [ 5860.661149] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5860.663337] Lustre: Skipped 2 previous similar messages [ 5860.772278] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5866.858495] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 5881.486833] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5881.652225] Lustre: 179446:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5882.103498] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5888.698675] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 5900.398154] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5908.242699] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5918.258130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5918.909281] Lustre: 181426:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5928.382486] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 5929.099913] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5929.205549] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5940.879980] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5949.922085] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5949.925462] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5949.940255] Lustre: Skipped 3 previous similar messages [ 5949.951469] Lustre: Skipped 4 previous similar messages [ 5956.131073] Lustre: server umount lustre-OST0000 complete [ 5964.263071] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5968.174967] Lustre: server umount lustre-MDT0000 complete [ 5970.408772] LustreError: 179459:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5970.428520] LustreError: 179459:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5972.155373] LustreError: 180247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788630339 with bad export cookie 9772264445155310619 [ 5972.165307] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5972.170429] LustreError: 180247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5972.425665] Lustre: server umount lustre-MDT0001 complete [ 5986.410944] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 5990.161533] Key type lgssc unregistered [ 5990.486687] LNet: 183283:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5990.489758] LNetError: 183283:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5990.501764] LNet: Removed LNI 192.168.202.127@tcp [ 5991.533192] Key type .llcrypt unregistered [ 5991.536908] Key type ._llcrypt unregistered [ 6015.625719] Key type ._llcrypt registered [ 6015.627409] Key type .llcrypt registered [ 6015.869716] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 6017.533738] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6017.616328] alg: No test for adler32 (adler32-zlib) [ 6019.146339] Lustre: Lustre: Build Version: 2.17.58_2_g69c2166 [ 6019.647985] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 6021.529437] Key type lgssc registered [ 6023.808251] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6036.162918] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6036.213251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6037.552617] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6044.594568] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 6047.721816] LustreError: 184697:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6047.762967] LustreError: 184697:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6052.845213] LustreError: 184696:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6057.954792] LustreError: 184697:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6059.677279] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6060.406435] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6067.524837] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 6077.062468] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6083.262979] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6091.428551] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6091.987590] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6099.689888] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 6107.402599] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 6108.382420] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6128.114422] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6128.133789] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6128.141048] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6128.156264] Lustre: Skipped 1 previous similar message [ 6133.217610] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6133.233414] Lustre: Skipped 1 previous similar message [ 6133.665682] Lustre: server umount lustre-OST0000 complete [ 6138.340557] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6138.344361] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6138.363512] Lustre: Skipped 2 previous similar messages [ 6138.377309] Lustre: Skipped 1 previous similar message [ 6143.457642] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6143.463224] Lustre: Skipped 1 previous similar message [ 6144.902752] Lustre: server umount lustre-MDT0000 complete [ 6148.577706] LustreError: 184693:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6148.595595] LustreError: 184693:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6149.427212] LustreError: 184678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788630516 with bad export cookie 9488603278921333915 [ 6149.428958] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6149.442648] LustreError: 184678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6149.887976] Lustre: server umount lustre-MDT0001 complete [ 6162.430589] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 6165.607318] Key type lgssc unregistered [ 6165.954852] LNet: 188257:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6165.961304] LNetError: 188257:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6165.973686] LNet: Removed LNI 192.168.202.127@tcp [ 6166.872742] Key type .llcrypt unregistered [ 6166.880940] Key type ._llcrypt unregistered [ 6194.303951] Key type ._llcrypt registered [ 6194.306021] Key type .llcrypt registered [ 6194.402485] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 6195.474574] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6195.527258] alg: No test for adler32 (adler32-zlib) [ 6196.618506] Lustre: Lustre: Build Version: 2.17.58_2_g69c2166 [ 6196.982443] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 6198.720223] Key type lgssc registered [ 6200.399123] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6209.854894] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6209.891306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6211.619564] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6217.549498] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 6221.798364] LustreError: 189667:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6221.819689] LustreError: 189667:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6226.923221] LustreError: 189668:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6227.484794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6227.999580] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6233.934237] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 6243.002575] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6250.999384] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6258.908980] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6259.408150] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6266.459704] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 6274.949368] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6281.916231] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6293.473789] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6293.491319] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6293.502111] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6297.079135] Lustre: lustre-OST0000: Not available for connect from 192.168.202.27@tcp (stopping) [ 6297.578409] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6297.589453] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6298.995493] Lustre: server umount lustre-OST0000 complete [ 6309.349736] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9832d0818800.ost_server_uuid 50 [ 6311.024306] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6327.269715] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6327.272446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6327.280460] Lustre: Skipped 1 previous similar message [ 6327.293528] Lustre: Skipped 3 previous similar messages [ 6328.225130] Lustre: server umount lustre-MDT0000 complete [ 6332.335291] LustreError: 189649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788630699 with bad export cookie 5964706743478134 [ 6332.346717] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6332.347713] LustreError: 189649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6332.792894] Lustre: server umount lustre-MDT0001 complete [ 6343.496263] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 13:51:48 (1788630708) [ 6350.498852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6350.955403] LustreError: 193604:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6351.082996] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6356.329545] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 6356.456540] LustreError: 193605:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6356.467926] LustreError: 193605:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6361.579650] LustreError: 193604:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6365.653351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6366.062119] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6371.438955] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 6380.193072] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6385.985755] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6393.840859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6394.362450] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6396.413680] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6401.636481] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 6410.085399] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6417.128673] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6417.453859] Lustre: lustre-OST0001: new disk, initializing [ 6417.458484] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6417.465461] Lustre: 196450:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6417.571117] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6423.814182] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 6424.142707] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6424.149279] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6424.263785] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6433.288913] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6436.321274] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6436.329564] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6436.350718] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6444.484940] Lustre: lustre-OST0000: Not available for connect from 192.168.202.27@tcp (stopping) [ 6444.496060] Lustre: Skipped 4 previous similar messages [ 6450.144994] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6450.265448] Lustre: server umount lustre-OST0000 complete [ 6454.726486] LustreError: 195430:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6457.614969] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6457.868317] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6463.164162] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6463.569949] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6473.160896] LustreError: 195430:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6473.176250] LustreError: 195430:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 6480.867357] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6480.875542] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6480.881389] LustreError: Skipped 2 previous similar messages [ 6480.899268] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6480.906194] Lustre: Skipped 2 previous similar messages [ 6480.926952] Lustre: Skipped 6 previous similar messages [ 6484.300931] Lustre: server umount lustre-OST0001 complete [ 6489.062561] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6489.081659] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6489.104400] Lustre: Skipped 1 previous similar message [ 6494.859281] Lustre: server umount lustre-MDT0000 complete [ 6499.350282] LustreError: 196449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788630866 with bad export cookie 5964706743479205 [ 6499.353539] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6499.364352] LustreError: 196449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6499.770150] Lustre: server umount lustre-MDT0001 complete [ 6529.223395] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 13:54:54 (1788630894) [ 6536.749627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6537.114729] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6537.144782] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6537.310888] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6542.620826] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 6552.297038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6552.387223] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6552.427941] Lustre: 199890:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6558.361169] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 6567.467794] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6574.421441] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6582.384037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6582.677796] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6591.418946] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 6592.819142] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6601.333703] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6608.561073] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6608.625809] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6608.837910] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6608.846362] Lustre: Skipped 2 previous similar messages [ 6615.015419] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 6622.489319] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6626.799145] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6626.810836] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6626.815481] Lustre: Skipped 1 previous similar message [ 6626.832810] Lustre: Skipped 4 previous similar messages [ 6631.684328] Lustre: server umount lustre-OST0000 complete [ 6631.912140] LustreError: 201741:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6631.924885] LustreError: 201741:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 6652.385238] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6652.397607] LustreError: Skipped 1 previous similar message [ 6654.195169] Lustre: server umount lustre-OST0001 complete [ 6660.589461] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6660.608562] Lustre: Skipped 2 previous similar messages [ 6660.620904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6660.627152] Lustre: Skipped 4 previous similar messages [ 6664.975710] Lustre: server umount lustre-MDT0000 complete [ 6669.215959] LustreError: 202778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788631036 with bad export cookie 5964706743480612 [ 6669.217573] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6669.221793] LustreError: 202778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6669.565162] Lustre: server umount lustre-MDT0001 complete [ 6698.536589] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 13:57:43 (1788631063) [ 6715.873524] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_hostid [ 6727.388596] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 6769.040503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6769.251565] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6769.271408] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6769.341996] Lustre: lustre-MDT0000: new disk, initializing [ 6769.420519] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6769.435179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6774.380294] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 6785.666332] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6785.775475] Lustre: 207988:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6785.866595] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6785.875282] Lustre: Skipped 1 previous similar message [ 6786.008110] Lustre: lustre-MDT0001: new disk, initializing [ 6786.240831] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6786.257627] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6791.140224] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 6803.291942] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6810.255932] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6818.206870] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6818.749468] Lustre: lustre-OST0000: new disk, initializing [ 6818.756137] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6818.762259] Lustre: 209956:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6827.013127] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 6828.153218] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6828.169733] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6828.359549] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6837.622431] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6841.467438] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6841.671731] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6846.521941] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6846.677678] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6848.994249] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6848.998825] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6849.009592] Lustre: Skipped 2 previous similar messages [ 6849.024403] Lustre: Skipped 3 previous similar messages [ 6863.329123] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6863.720990] Lustre: server umount lustre-OST0000 complete [ 6867.938140] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6867.948923] LustreError: Skipped 1 previous similar message [ 6882.272966] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6882.568651] Lustre: server umount lustre-MDT0000 complete [ 6884.834996] LustreError: 207998:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6884.851137] LustreError: 207998:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [ 6886.724863] LustreError: 207980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788631253 with bad export cookie 5964706743482166 [ 6886.731711] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6886.738321] LustreError: 207980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6887.200666] Lustre: server umount lustre-MDT0001 complete [ 6897.201288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6902.870430] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 6911.925794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6917.097476] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 6926.138839] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6932.987204] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6940.362451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6948.628155] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 6956.600647] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6962.105649] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6962.335890] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6967.430928] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6967.685338] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6972.389086] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6976.218325] Lustre: server umount lustre-OST0000 complete [ 6983.053755] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6987.845714] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7017.909594] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7025.787787] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 7035.258654] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7037.698336] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7037.717715] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7037.718679] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7037.904452] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7048.166131] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7048.170235] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7048.181755] LustreError: Skipped 1 previous similar message [ 7048.192571] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7048.197507] Lustre: Skipped 5 previous similar messages [ 7048.236265] Lustre: Skipped 14 previous similar messages [ 7049.359424] Lustre: server umount lustre-OST0000 complete [ 7067.104145] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7067.421788] Lustre: server umount lustre-MDT0000 complete [ 7068.660634] LustreError: 212187:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7068.719658] LustreError: 212187:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 7072.955530] LustreError: 212167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788631439 with bad export cookie 5964706743483055 [ 7072.973094] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7073.334620] Lustre: server umount lustre-MDT0001 complete [ 7083.194710] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 14:04:08 (1788631448) [ 7091.099589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7091.698775] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7091.706470] Lustre: Skipped 6 previous similar messages [ 7097.068875] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 7106.304129] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7112.577711] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 7121.546748] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7128.251190] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7135.683878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7143.659503] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 7152.156767] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7157.575287] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7157.780497] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7163.190303] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7163.411237] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7170.692166] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7170.897763] Lustre: lustre-OST0001: new disk, initializing [ 7170.903582] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7170.915826] Lustre: 221526:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7177.315647] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 7178.838147] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7178.845033] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7178.939644] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7186.766398] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7191.838846] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7192.068850] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7197.185780] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7197.435655] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7199.727544] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7199.743479] LustreError: Skipped 1 previous similar message [ 7205.707966] Lustre: server umount lustre-OST0001 complete [ 7212.890500] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7213.048477] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7218.021331] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7218.288569] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7247.512548] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7252.977228] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7253.009118] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7253.031596] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7253.041549] Lustre: Skipped 1 previous similar message [ 7253.870978] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 7262.477840] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7264.576992] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7281.120145] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7281.277289] Lustre: server umount lustre-OST0001 complete [ 7305.954725] Lustre: server umount lustre-OST0000 complete [ 7311.330254] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7311.340024] Lustre: Skipped 8 previous similar messages [ 7311.344831] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7311.352152] Lustre: Skipped 20 previous similar messages [ 7316.178075] Lustre: server umount lustre-MDT0000 complete [ 7320.913284] LustreError: 218138:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788631687 with bad export cookie 5964706743484014 [ 7320.919477] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7320.925415] LustreError: 218138:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 7335.393241] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7335.917914] Lustre: server umount lustre-MDT0001 complete [ 7362.546214] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 14:08:47 (1788631727) [ 7371.117193] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7371.900463] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7371.939745] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7378.250307] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 7388.476078] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7388.669601] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7388.698678] Lustre: 226777:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7393.343726] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 7403.433362] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7411.219204] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7419.060278] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7419.315952] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7427.793294] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 7437.916391] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7448.472586] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7448.585223] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7455.421796] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 7464.004843] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7465.920396] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7471.874356] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7472.109981] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7477.095097] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7477.257214] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7483.867542] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9832d081b800.ost_server_uuid 50 [ 7485.900553] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9832d081b800.ost_server_uuid in IDLE state after 0 sec [ 7487.067929] Lustre: Permanently deactivating lustre-OST0001 [ 7487.085827] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7491.037804] Lustre: Permanently reactivating lustre-OST0001 [ 7497.206297] LustreError: lustre-OST0001-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 7497.221853] LustreError: Skipped 5 previous similar messages [ 7502.169474] Lustre: server umount lustre-OST0001 complete [ 7505.897087] LustreError: 228625:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7505.924429] LustreError: 228625:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 38 previous similar messages [ 7512.708628] Lustre: server umount lustre-OST0000 complete [ 7523.082622] Lustre: server umount lustre-MDT0000 complete [ 7527.532256] LustreError: 226768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788631894 with bad export cookie 5964706743485309 [ 7527.553704] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7527.821516] Lustre: server umount lustre-MDT0001 complete [ 7554.183835] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 14:11:59 (1788631919) [ 7562.509714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7562.868693] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7562.904389] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7562.911600] Lustre: Skipped 5 previous similar messages [ 7568.122367] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 7577.333766] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7577.427051] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7577.459779] Lustre: 233476:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7583.244711] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 7591.925063] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7597.699533] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7606.364685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7606.573957] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7606.601408] Lustre: Permanently deactivating lustre-OST0000 [ 7606.615352] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7606.833539] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7606.851146] Lustre: Skipped 10 previous similar messages [ 7613.853945] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 7616.085578] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7616.179927] LustreError: 235820:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -108 [ 7621.920687] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7629.608270] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7629.757490] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7636.182799] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7639.006993] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 7646.663792] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7648.692265] Lustre: Permanently reactivating lustre-OST0000 [ 7648.715034] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7648.720685] Lustre: Skipped 2 previous similar messages [ 7657.088684] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 7657.116929] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 41 seconds. I think it's dead, and I am evicting it. exp 00000000e02f30ab, cur 1788632024 deadline 1788632013 last 1788631983 [ 7657.210059] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7667.397105] Lustre: server umount lustre-OST0001 complete [ 7677.713677] Lustre: server umount lustre-OST0000 complete [ 7687.993093] Lustre: server umount lustre-MDT0000 complete [ 7691.590492] LustreError: 235357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788632058 with bad export cookie 5964706743486982 [ 7691.611263] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7691.618257] LustreError: 235357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 7691.967260] Lustre: server umount lustre-MDT0001 complete [ 7701.659176] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 7704.314547] Key type lgssc unregistered [ 7704.573306] LNet: 238369:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7704.581134] LNetError: 238369:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7705.638183] LNet: Removed LNI 192.168.202.127@tcp [ 7706.549179] Key type .llcrypt unregistered [ 7706.550494] Key type ._llcrypt unregistered [ 7732.756208] Key type ._llcrypt registered [ 7732.765697] Key type .llcrypt registered [ 7732.951874] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 7734.461806] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7734.470863] alg: No test for adler32 (adler32-zlib) [ 7735.441397] Lustre: Lustre: Build Version: 2.17.58_2_g69c2166 [ 7735.576970] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 7737.192223] Key type lgssc registered [ 7738.272441] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7750.923290] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7750.978898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7752.782167] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7759.460171] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 7762.920653] LustreError: 239836:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7762.935991] LustreError: 239836:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7768.040901] LustreError: 239837:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7770.427529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7770.580487] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7770.622450] Lustre: 239826:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7770.628242] Lustre: Permanently deactivating lustre-MDT0001 [ 7770.658674] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7771.196189] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7776.485983] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 7780.883768] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7785.320845] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7791.528701] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7799.503143] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7799.780336] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7806.665980] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 7815.150896] LustreError: 241679:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7815.179665] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7816.307556] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7820.264561] LustreError: 241681:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7822.381173] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7824.459429] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7824.722825] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7830.033791] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7830.039383] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7832.597946] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 7845.488121] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7850.095185] Lustre: Permanently reactivating lustre-MDT0001 [ 7850.140028] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7850.143215] Lustre: Skipped 2 previous similar messages [ 7854.297982] Lustre: lustre-MDT0001: Client f8a18be3-b02b-430a-b979-1a4978e81b90 (at 192.168.202.27@tcp) reconnecting [ 7855.716209] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7855.730262] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7855.738761] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7855.747703] LustreError: 243417:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 7855.752518] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7855.761353] LustreError: 243417:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 7855.761482] LustreError: 243417:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.127@tcp: cfg command failed: rc = -108 [ 7855.761844] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 7855.761844] [ 7855.762130] LustreError: 239828:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 7860.496175] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7862.906444] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7870.093207] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7870.299401] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7875.543075] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7875.878919] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7879.203522] Lustre: Permanently deactivating lustre-MDT0001 [ 7879.221398] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7879.224931] Lustre: Skipped 2 previous similar messages [ 7885.987793] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7896.546194] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7896.556936] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7896.568942] Lustre: Skipped 3 previous similar messages [ 7896.583080] Lustre: Skipped 3 previous similar messages [ 7901.615194] Lustre: server umount lustre-MDT0000 complete [ 7901.668919] LustreError: 240581:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7901.695481] LustreError: 240581:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 7906.424218] LustreError: 239818:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788632273 with bad export cookie 8329556872237706350 [ 7906.433167] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7906.438885] LustreError: 239818:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7907.079782] Lustre: server umount lustre-MDT0001 complete [ 7913.637540] Lustre: server umount lustre-OST0000 complete [ 7926.154952] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 14:18:11 (1788632291) [ 7938.160545] Lustre: server umount lustre-OST0001 complete [ 7946.770321] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_hostid [ 7959.979454] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 8000.711569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8001.113510] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8001.119589] Lustre: Skipped 2 previous similar messages [ 8001.132407] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8001.174960] Lustre: lustre-MDT0000: new disk, initializing [ 8001.221256] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8001.231092] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8006.128727] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 8017.823270] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8017.935810] Lustre: 248097:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8018.000716] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8018.004504] Lustre: Skipped 1 previous similar message [ 8018.152420] Lustre: lustre-MDT0001: new disk, initializing [ 8018.262947] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8018.301244] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8018.316260] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8024.015799] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 8034.124417] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8040.771271] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8048.523880] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8048.886924] Lustre: lustre-OST0000: new disk, initializing [ 8048.891029] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8048.913807] Lustre: 250073:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8049.035268] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8056.070950] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 8058.498961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8058.507651] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8058.657402] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8067.023930] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8069.331404] LustreError: 249461:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8078.406788] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8078.607515] Lustre: lustre-OST0001: new disk, initializing [ 8078.610285] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8078.614779] Lustre: 251186:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8079.424612] LustreError: 249461:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8079.445427] LustreError: 250947:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8085.005906] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 8085.126776] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8085.141614] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8085.290918] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8089.497367] LustreError: 250947:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8096.028952] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8098.266965] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8100.834335] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8100.836817] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8100.849096] LustreError: Skipped 1 previous similar message [ 8100.849960] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8100.867186] Lustre: Skipped 1 previous similar message [ 8100.880219] Lustre: Skipped 1 previous similar message [ 8101.860074] Lustre: lustre-OST0001: Not available for connect from 192.168.202.27@tcp (stopping) [ 8105.954305] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8105.963211] Lustre: Skipped 1 previous similar message [ 8106.333679] Lustre: server umount lustre-OST0001 complete [ 8106.984791] LustreError: 250056:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8106.994933] LustreError: 250056:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8125.409854] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8125.415829] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8125.431122] LustreError: Skipped 1 previous similar message [ 8125.481176] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8126.462609] LustreError: 250536:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8126.499512] LustreError: 250536:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 8129.664695] Lustre: server umount lustre-OST0000 complete [ 8136.161463] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8136.173125] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8136.185859] Lustre: Skipped 1 previous similar message [ 8136.189685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8136.196457] Lustre: Skipped 3 previous similar messages [ 8140.514403] Lustre: server umount lustre-MDT0000 complete [ 8144.577072] LustreError: 248092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788632511 with bad export cookie 8329556872237709164 [ 8144.583667] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8144.592293] LustreError: 248092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8151.046764] Lustre: server umount lustre-MDT0001 complete [ 8164.043521] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 8167.085608] Key type lgssc unregistered [ 8167.365781] LNet: 253216:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8167.375864] LNetError: 253216:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8167.395418] LNet: Removed LNI 192.168.202.127@tcp [ 8168.441687] Key type .llcrypt unregistered [ 8168.444611] Key type ._llcrypt unregistered [ 8211.037493] Key type ._llcrypt registered [ 8211.039828] Key type .llcrypt registered [ 8211.174535] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 8213.100738] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8213.161573] alg: No test for adler32 (adler32-zlib) [ 8214.311754] Lustre: Lustre: Build Version: 2.17.58_2_g69c2166 [ 8214.649842] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 8216.328174] Key type lgssc registered [ 8217.693821] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8226.468029] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8226.506632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8227.766494] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8227.796045] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8227.998576] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8233.033673] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 8242.325283] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8242.408608] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8242.468546] Lustre: 255425:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8242.800117] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8247.570879] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 8255.575775] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8261.187825] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8268.711711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8268.997023] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8269.209115] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8275.927342] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 8276.622672] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8283.572922] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8337.894717] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8337.911695] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8337.935898] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8338.404824] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8342.481376] Lustre: server umount lustre-OST0000 complete [ 8347.006631] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8362.048503] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8362.325334] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8363.377832] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8363.394700] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8368.835424] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 8376.921531] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8378.730119] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8378.743251] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8419.811050] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8419.812460] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8419.824984] LustreError: Skipped 1 previous similar message [ 8419.826063] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8419.826067] Lustre: Skipped 2 previous similar messages [ 8419.870468] Lustre: Skipped 2 previous similar messages [ 8422.265685] Lustre: server umount lustre-OST0000 complete [ 8424.929354] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8424.936488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8424.945365] Lustre: Skipped 1 previous similar message [ 8424.956027] Lustre: Skipped 2 previous similar messages [ 8430.051806] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8430.058225] Lustre: Skipped 1 previous similar message [ 8435.170935] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8435.189317] Lustre: Skipped 1 previous similar message [ 8439.264246] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8439.610651] Lustre: server umount lustre-MDT0000 complete [ 8445.196477] LustreError: 255415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788632812 with bad export cookie 7963969182038266554 [ 8445.203446] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8445.229235] LustreError: 255415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8445.927479] Lustre: server umount lustre-MDT0001 complete [ 8461.772211] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 8465.202397] Key type lgssc unregistered [ 8465.404814] LNet: 260849:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8465.410815] LNetError: 260849:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8465.435640] LNet: Removed LNI 192.168.202.127@tcp [ 8466.424913] Key type .llcrypt unregistered [ 8466.426817] Key type ._llcrypt unregistered [ 8495.546593] Key type ._llcrypt registered [ 8495.551446] Key type .llcrypt registered [ 8495.689652] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 8496.724936] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8496.871790] alg: No test for adler32 (adler32-zlib) [ 8498.076631] Lustre: Lustre: Build Version: 2.17.58_2_g69c2166 [ 8498.445358] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 8500.136228] Key type lgssc registered [ 8501.207767] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8509.417116] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8509.433331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8510.988856] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8516.076793] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 8520.166464] LustreError: 262263:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8520.178339] LustreError: 262263:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8525.242052] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8525.282975] LustreError: 262264:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8525.666466] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8531.350832] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 8540.225483] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8546.905126] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8554.698246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8555.116018] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8556.148620] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8562.498218] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 8571.260294] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8601.571556] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8601.587709] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8601.595164] Lustre: Skipped 1 previous similar message [ 8601.599213] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8603.770357] Lustre: server umount lustre-OST0000 complete [ 8611.817234] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8611.821456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8611.833535] Lustre: Skipped 1 previous similar message [ 8611.838210] Lustre: Skipped 1 previous similar message [ 8614.499333] Lustre: server umount lustre-MDT0000 complete [ 8616.942809] LustreError: 262258:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8616.965368] LustreError: 262258:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8618.941652] LustreError: 262244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788632985 with bad export cookie 2160395678947193915 [ 8618.961440] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8619.425711] Lustre: server umount lustre-MDT0001 complete [ 8631.346801] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 8634.243280] Key type lgssc unregistered [ 8634.551375] LNet: 266569:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8634.558854] LNetError: 266569:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8634.584967] LNet: Removed LNI 192.168.202.127@tcp [ 8636.014216] Key type .llcrypt unregistered [ 8636.019365] Key type ._llcrypt unregistered [ 8656.071282] Key type ._llcrypt registered [ 8656.074934] Key type .llcrypt registered [ 8656.241638] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 8658.178882] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8658.330079] alg: No test for adler32 (adler32-zlib) [ 8660.223879] Lustre: Lustre: Build Version: 2.17.58_2_g69c2166 [ 8660.623969] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 8662.489820] Key type lgssc registered [ 8664.070990] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8673.889722] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8673.902465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8675.630156] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8681.870754] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 8685.045771] LustreError: 267790:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8685.060409] LustreError: 267790:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8690.152199] LustreError: 267791:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8691.845401] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8692.260419] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8697.867753] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 8707.415671] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8714.547645] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8723.205829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8723.584823] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8730.379593] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 8737.851905] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8745.091810] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8760.803165] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8760.809375] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8760.822744] Lustre: Skipped 1 previous similar message [ 8760.844502] Lustre: Skipped 1 previous similar message [ 8763.088251] Lustre: server umount lustre-OST0000 complete [ 8768.995715] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8769.004433] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8769.031327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8771.046220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8771.048568] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8776.168180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8776.178589] Lustre: Skipped 2 previous similar messages [ 8780.258393] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8780.263663] Lustre: Skipped 1 previous similar message [ 8782.304259] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8782.517916] Lustre: server umount lustre-MDT0000 complete [ 8785.382072] LustreError: 269632:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8785.410996] LustreError: 269632:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8786.965993] LustreError: 270739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788633153 with bad export cookie 1140380316000829040 [ 8786.985875] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8787.373223] Lustre: server umount lustre-MDT0001 complete [ 8799.009085] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 8802.447428] Key type lgssc unregistered [ 8802.768704] LNet: 271492:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8802.777621] LNetError: 271492:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8802.799505] LNet: Removed LNI 192.168.202.127@tcp [ 8803.897830] Key type .llcrypt unregistered [ 8803.899457] Key type ._llcrypt unregistered [ 8828.779712] Key type ._llcrypt registered [ 8828.782237] Key type .llcrypt registered [ 8828.874965] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 8830.316484] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8830.343379] alg: No test for adler32 (adler32-zlib) [ 8831.421419] Lustre: Lustre: Build Version: 2.17.58_2_g69c2166 [ 8831.629101] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 8833.320200] Key type lgssc registered [ 8834.560146] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8843.538325] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8843.562049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8845.165263] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8850.292822] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 8855.529788] LustreError: 272906:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8855.544484] LustreError: 272906:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8860.644824] LustreError: 272907:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8860.710311] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8861.177418] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8866.224721] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 8875.767639] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8882.457573] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8890.297606] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8890.702523] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8891.781149] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8897.891930] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 8905.944562] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8936.929736] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8936.934391] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8936.965435] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8936.986990] Lustre: Skipped 1 previous similar message [ 8937.005719] Lustre: Skipped 1 previous similar message [ 8939.936802] Lustre: server umount lustre-OST0000 complete [ 8947.177702] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8947.195950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8950.689608] Lustre: server umount lustre-MDT0000 complete [ 8952.312094] LustreError: 272906:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8952.332084] LustreError: 272906:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8955.365835] LustreError: 272886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788633322 with bad export cookie 11297075652415619664 [ 8955.386840] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8955.771984] Lustre: server umount lustre-MDT0001 complete [ 8970.362668] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 8973.901256] Key type lgssc unregistered [ 8974.364825] LNet: 277212:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8974.377422] LNetError: 277212:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8974.407785] LNet: Removed LNI 192.168.202.127@tcp [ 8975.432498] Key type .llcrypt unregistered [ 8975.439104] Key type ._llcrypt unregistered [ 8995.022687] Key type ._llcrypt registered [ 8995.025313] Key type .llcrypt registered [ 8995.159858] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 8996.572774] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8996.626921] alg: No test for adler32 (adler32-zlib) [ 8997.839742] Lustre: Lustre: Build Version: 2.17.58_2_g69c2166 [ 8998.224899] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 8999.952355] Key type lgssc registered [ 9001.398496] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9010.710363] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9010.773647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9012.512883] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9017.732983] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 9022.965211] LustreError: 278448:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9022.992743] LustreError: 278448:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9027.902257] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9028.075719] LustreError: 278450:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9028.456147] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9033.286576] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 9041.132198] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9047.977559] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9055.728835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9056.157260] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9057.160241] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 9065.614692] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 9076.597263] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9093.101469] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9093.116915] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9093.148474] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9094.125212] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9094.126610] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9094.134232] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9094.156548] Lustre: Skipped 1 previous similar message [ 9096.433981] Lustre: server umount lustre-OST0000 complete [ 9104.373336] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9104.377100] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9104.396210] Lustre: Skipped 1 previous similar message [ 9104.413602] Lustre: Skipped 1 previous similar message [ 9107.455486] Lustre: server umount lustre-MDT0000 complete [ 9109.474893] LustreError: 278460:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9109.493782] LustreError: 278460:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9111.418155] LustreError: 281399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788633478 with bad export cookie 3727830931785677736 [ 9111.424803] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9111.432522] LustreError: 281399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9112.210343] Lustre: server umount lustre-MDT0001 complete [ 9126.670270] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 9128.799287] Key type lgssc unregistered [ 9129.081808] LNet: 282150:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9129.094695] LNetError: 282150:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9129.113729] LNet: Removed LNI 192.168.202.127@tcp [ 9129.806858] Key type .llcrypt unregistered [ 9129.813659] Key type ._llcrypt unregistered [ 9147.905453] Key type ._llcrypt registered [ 9147.908110] Key type .llcrypt registered [ 9148.000251] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 9166.263229] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_hostid [ 9180.488296] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 9181.592821] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9181.810107] alg: No test for adler32 (adler32-zlib) [ 9182.874471] Lustre: Lustre: Build Version: 2.17.58_2_g69c2166 [ 9183.073548] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 9184.752196] Key type lgssc registered [ 9185.784957] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9223.118705] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9223.151826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9224.317707] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9224.343105] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9224.413100] Lustre: lustre-MDT0000: new disk, initializing [ 9224.570391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9224.626873] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9229.021116] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 9240.455298] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9240.835362] Lustre: 285710:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9241.061383] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9241.071411] Lustre: Skipped 1 previous similar message [ 9241.258513] Lustre: lustre-MDT0001: new disk, initializing [ 9241.486939] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9241.605451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9241.620209] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9248.963146] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 9259.716730] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9266.558143] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9274.875459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9275.437939] Lustre: lustre-OST0000: new disk, initializing [ 9275.443150] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9275.451331] Lustre: 287678:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9275.662857] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9283.739244] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9283.753187] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9283.969726] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9285.351184] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 9298.713982] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9304.534348] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9304.784442] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9309.983752] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9310.278856] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9314.788399] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9314.797169] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9314.803665] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9314.828834] Lustre: Skipped 1 previous similar message [ 9314.851976] Lustre: Skipped 1 previous similar message [ 9318.885819] Lustre: server umount lustre-OST0000 complete [ 9323.492878] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9323.499929] LustreError: Skipped 1 previous similar message [ 9323.503332] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9323.528457] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9325.026313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9325.029302] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9329.803906] Lustre: server umount lustre-MDT0000 complete [ 9333.656208] LustreError: 285701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788633700 with bad export cookie 7798713793460900891 [ 9333.658305] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9333.673778] LustreError: 285701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9333.929937] Lustre: server umount lustre-MDT0001 complete [ 9344.161516] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 14:41:48 (1788633708) [ 9352.100071] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9352.441756] LustreError: 290095:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9352.460672] LustreError: 290095:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9352.519255] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9357.821919] LustreError: 290096:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9359.264570] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 9362.921180] LustreError: 290096:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9368.053168] LustreError: 290095:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9371.944332] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9372.671843] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9379.100535] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 9388.266347] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9394.502304] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9401.800886] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9402.245710] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9410.204848] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 9418.036461] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9433.070757] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9433.074711] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9433.082152] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9433.082165] Lustre: Skipped 1 previous similar message [ 9433.103699] Lustre: Skipped 1 previous similar message [ 9433.131205] LustreError: Skipped 1 previous similar message [ 9438.193757] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9438.206907] Lustre: Skipped 2 previous similar messages [ 9438.673643] Lustre: server umount lustre-OST0000 complete [ 9443.298171] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9445.243213] Lustre: server umount lustre-MDT0000 complete [ 9448.418228] LustreError: 290823:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9448.441100] LustreError: 290823:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9449.331209] LustreError: 290076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788633816 with bad export cookie 7798713793460901780 [ 9449.334550] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9449.340906] LustreError: 290076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9449.749540] Lustre: server umount lustre-MDT0001 complete [ 9462.169381] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 9465.112584] Key type lgssc unregistered [ 9465.413467] LNet: 293653:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9465.419349] LNetError: 293653:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9465.434206] LNet: Removed LNI 192.168.202.127@tcp [ 9466.052232] Key type .llcrypt unregistered [ 9466.054328] Key type ._llcrypt unregistered [ 9499.106206] Key type ._llcrypt registered [ 9499.108593] Key type .llcrypt registered [ 9499.240833] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_hostid [ 9513.951925] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 9515.664946] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9515.728989] alg: No test for adler32 (adler32-zlib) [ 9517.005747] Lustre: Lustre: Build Version: 2.17.58_2_g69c2166 [ 9517.285170] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 9518.944226] Key type lgssc registered [ 9520.179208] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9564.200250] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9564.262747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9565.627755] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9565.658243] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9565.725787] Lustre: lustre-MDT0000: new disk, initializing [ 9565.805529] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9565.819368] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9570.845900] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 9582.319919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9582.466331] Lustre: 297479:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9582.566112] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9582.573471] Lustre: Skipped 1 previous similar message [ 9582.684479] Lustre: lustre-MDT0001: new disk, initializing [ 9582.860836] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9582.901685] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9582.915844] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9587.915719] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 9596.879286] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9602.461524] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9609.195668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9609.582333] Lustre: lustre-OST2710: new disk, initializing [ 9609.584765] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9609.592442] Lustre: 299446:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9609.718108] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9615.942477] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9615.955202] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9616.090520] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9616.456774] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 9626.646847] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9633.776428] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9633.994417] Lustre: lustre-OST03e8: new disk, initializing [ 9633.997690] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9634.003680] Lustre: 300510:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9634.088669] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9640.137773] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 9641.580710] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9641.600283] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9641.705868] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9647.443676] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9655.676448] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9655.887273] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9661.334507] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9661.547889] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9666.703068] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9666.911224] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9671.963963] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9672.216925] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9677.659414] Lustre: server umount lustre-MDT0000 complete [ 9677.800392] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9677.813031] Lustre: Skipped 2 previous similar messages [ 9681.065956] LustreError: 299445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788634048 with bad export cookie 16427896807531262293 [ 9681.080719] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9681.439904] Lustre: server umount lustre-MDT0001 complete [ 9695.660339] Lustre: server umount lustre-OST2710 complete [ 9698.660576] Lustre: 295277:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788634049/real 1788634049] req@0000000001f9a451 x1875518563617280/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788634065 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9698.687485] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9699.540851] Lustre: server umount lustre-OST03e8 complete [ 9718.908574] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_hostid [ 9730.044386] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 9768.329840] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9768.629572] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9768.663391] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9768.759869] Lustre: lustre-MDT0000: new disk, initializing [ 9768.819652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9768.832920] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9773.718880] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 9783.828255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9783.995610] Lustre: 305760:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9784.061194] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9784.069460] Lustre: Skipped 1 previous similar message [ 9784.198688] Lustre: lustre-MDT0001: new disk, initializing [ 9784.343823] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9784.415437] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9784.425302] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9789.797171] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 9800.887300] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9806.746561] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9815.167449] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9815.427911] Lustre: lustre-OST0000: new disk, initializing [ 9815.431893] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9815.443076] Lustre: 307728:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9815.558296] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9822.332830] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9822.353780] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9822.526355] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9822.758396] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 9835.422917] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9842.604581] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9842.978942] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9849.234116] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9849.458320] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9853.413849] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9853.419021] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9853.423071] LustreError: Skipped 1 previous similar message [ 9853.427249] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9853.432054] Lustre: Skipped 1 previous similar message [ 9853.451790] Lustre: Skipped 1 previous similar message [ 9857.614719] Lustre: server umount lustre-OST0000 complete [ 9863.287803] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9863.293717] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9863.319124] Lustre: Skipped 1 previous similar message [ 9868.771120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9868.788713] Lustre: Skipped 2 previous similar messages [ 9869.848993] Lustre: server umount lustre-MDT0000 complete [ 9873.848480] LustreError: 305750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788634240 with bad export cookie 16427896807531263973 [ 9873.850260] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9873.855469] LustreError: 305750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 9874.143581] Lustre: server umount lustre-MDT0001 complete [ 9883.019645] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 14:50:48 (1788634248) [ 9884.738994] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9886.294932] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 14:50:52 (1788634252) [ 9890.600409] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 9919.634737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9919.961506] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9919.979234] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9920.155087] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9925.204021] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 9933.483528] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9933.614981] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9933.665415] Lustre: 311587:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9937.501116] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 9957.749362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9957.964589] Lustre: Denying initial registration attempt from nid 192.168.202.127@tcp, specified as failover [ 9957.978075] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9957.983361] LustreError: 313041:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -99 [ 9957.989462] LustreError: 313041:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 9957.993197] LustreError: 313041:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9958.057663] Lustre: server umount lustre-OST0000 complete [ 9958.067645] LustreError: 313041:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9968.102325] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9968.106861] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9968.127800] Lustre: Skipped 1 previous similar message [ 9968.144073] Lustre: Skipped 1 previous similar message [ 9971.878959] Lustre: server umount lustre-MDT0000 complete [ 9973.218468] LustreError: 311598:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9973.233646] LustreError: 311598:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 9975.699764] LustreError: 311578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788634342 with bad export cookie 16427896807531264855 [ 9975.708417] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9975.708848] LustreError: 311578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9976.044640] Lustre: server umount lustre-MDT0001 complete [ 9989.862819] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 9992.865508] Key type lgssc unregistered [ 9993.160910] LNet: 314161:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9993.170920] LNetError: 314161:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9993.200548] LNet: Removed LNI 192.168.202.127@tcp [ 9994.368611] Key type .llcrypt unregistered [ 9994.370389] Key type ._llcrypt unregistered [10008.032531] Key type ._llcrypt registered [10008.035409] Key type .llcrypt registered [10008.164601] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [10009.291730] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10009.397292] alg: No test for adler32 (adler32-zlib) [10010.533881] Lustre: Lustre: Build Version: 2.17.58_2_g69c2166 [10010.809918] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [10012.537074] Key type lgssc registered [10013.781854] Lustre: Echo OBD driver; http://www.lustre.org/ [10059.296954] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [10069.198590] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10069.223622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10070.588562] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10070.626556] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10070.853370] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10075.343249] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [10084.337740] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10084.439321] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10084.465689] Lustre: 316925:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10084.812887] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10089.447726] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [10097.029553] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10102.712914] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10122.676209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10122.889806] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10123.099369] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10129.518534] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [10137.355947] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10140.132148] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10140.136546] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10140.144657] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10140.647957] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10140.657125] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10140.671543] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10145.761419] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10150.885507] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10150.894275] Lustre: Skipped 1 previous similar message [10153.952138] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10154.323192] Lustre: server umount lustre-OST0000 complete [10159.079506] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10159.080981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10159.106161] Lustre: Skipped 1 previous similar message [10159.119906] Lustre: Skipped 2 previous similar messages [10164.675778] Lustre: server umount lustre-MDT0000 complete [10168.662507] LustreError: 316916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788634535 with bad export cookie 12689575790508528479 [10168.665484] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10168.672637] LustreError: 316916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10169.055721] Lustre: server umount lustre-MDT0001 complete [10181.175115] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [10184.588187] Key type lgssc unregistered [10185.029773] LNet: 320498:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10185.038932] LNetError: 320498:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10185.099676] LNet: Removed LNI 192.168.202.127@tcp [10186.116448] Key type .llcrypt unregistered [10186.121167] Key type ._llcrypt unregistered [10214.565756] Key type ._llcrypt registered [10214.568124] Key type .llcrypt registered [10214.665878] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [10216.146307] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10216.185188] alg: No test for adler32 (adler32-zlib) [10217.450792] Lustre: Lustre: Build Version: 2.17.58_2_g69c2166 [10217.846718] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [10219.592789] Key type lgssc registered [10220.973772] Lustre: Echo OBD driver; http://www.lustre.org/ [10228.721922] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10228.745780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10230.202147] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10234.791103] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [10240.483211] LustreError: 321916:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10240.498057] LustreError: 321916:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10243.373257] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10243.741216] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10248.197199] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [10255.840726] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10262.007911] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10271.341569] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10271.796795] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10279.800984] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [10289.520568] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10298.858446] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10298.862466] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10298.872987] Lustre: Skipped 2 previous similar messages [10298.883505] Lustre: Skipped 2 previous similar messages [10301.921688] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.27@tcp (stopping) [10303.972402] LustreError: 321912:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10303.998497] LustreError: 321912:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10304.088809] Lustre: server umount lustre-MDT0000 complete [10307.024367] LustreError: 321913:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10308.063969] LustreError: 324500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788634675 with bad export cookie 10183932520738004011 [10308.072419] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10308.075548] LustreError: 324500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10309.090622] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10309.100553] LustreError: 321912:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10309.109565] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10312.201274] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.27@tcp (stopping) [10313.130321] LustreError: 321912:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10313.158267] LustreError: 321912:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10317.259280] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.27@tcp (stopping) [10317.269382] Lustre: Skipped 1 previous similar message [10322.374142] LustreError: 321911:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10322.400112] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10322.410655] LustreError: 321911:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10322.627832] Lustre: server umount lustre-MDT0001 complete [10330.974938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10340.279831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10353.538217] LustreError: 323748:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10353.640174] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10353.701539] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10358.930796] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [10367.351350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10367.705736] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10372.368994] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [10374.130494] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10380.170827] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10381.639070] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10387.844768] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10389.358601] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10394.603660] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10394.612881] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10394.615976] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10394.616026] Lustre: Skipped 2 previous similar messages [10394.632146] Lustre: Skipped 1 previous similar message [10400.716057] Lustre: server umount lustre-OST0000 complete [10408.937876] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10408.945587] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10411.407349] Lustre: server umount lustre-MDT0000 complete [10415.074089] LustreError: 326758:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10415.100103] LustreError: 326758:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10415.725366] LustreError: 321898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788634782 with bad export cookie 10183932520738011683 [10415.731820] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10415.732048] LustreError: 321898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10416.078719] Lustre: server umount lustre-MDT0001 complete [10428.910414] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [10432.190791] Key type lgssc unregistered [10432.537720] LNet: 328185:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10432.546871] LNetError: 328185:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10433.574986] LNet: Removed LNI 192.168.202.127@tcp [10434.919399] Key type .llcrypt unregistered [10434.921446] Key type ._llcrypt unregistered [10462.622858] Key type ._llcrypt registered [10462.624822] Key type .llcrypt registered [10462.832102] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [10464.169436] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10464.314481] alg: No test for adler32 (adler32-zlib) [10465.699909] Lustre: Lustre: Build Version: 2.17.58_2_g69c2166 [10466.082436] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [10467.795053] Key type lgssc registered [10469.580595] Lustre: Echo OBD driver; http://www.lustre.org/ [10479.192671] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10479.209790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10480.818548] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10486.836629] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [10490.339416] LustreError: 329606:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10490.361029] LustreError: 329606:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10495.463237] LustreError: 329607:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10496.730511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10497.337391] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10502.765699] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [10511.650287] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10519.436668] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10527.211595] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10527.220048] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10527.245065] Lustre: Skipped 1 previous similar message [10527.262806] Lustre: Skipped 1 previous similar message [10532.322524] LustreError: 329606:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10532.339867] LustreError: 329606:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10532.541769] Lustre: server umount lustre-MDT0000 complete [10536.591243] LustreError: 329589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788634903 with bad export cookie 13440376473986284445 [10536.600782] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10536.992913] Lustre: server umount lustre-MDT0001 complete [10553.394626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10553.791518] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10553.827646] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10553.995033] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10558.660408] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [10569.051488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10569.232417] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10569.300826] Lustre: 332209:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10569.731125] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10576.175817] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [10587.196869] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10595.039854] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10605.161249] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10605.592860] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10605.616448] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10605.639350] LustreError: 334094:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [10605.658708] LustreError: 334094:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [10605.671826] LustreError: 334094:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10605.835538] Lustre: server umount lustre-OST0000 complete [10605.837666] LustreError: 334094:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10620.842946] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10621.267557] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10621.666424] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10630.146798] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [10630.388186] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10638.849520] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10648.592798] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10648.738150] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10648.841957] Lustre: lustre-OST0001: new disk, initializing [10648.846390] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10648.852560] Lustre: 335635:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10648.980216] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10656.899152] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10656.910409] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10657.063066] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10657.171169] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [10664.514817] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10672.630298] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10672.637182] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10672.649218] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10672.652979] LustreError: Skipped 1 previous similar message [10677.743529] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10677.751293] Lustre: Skipped 2 previous similar messages [10682.337538] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10682.460505] Lustre: server umount lustre-OST0001 complete [10682.854815] LustreError: 334561:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10682.877346] LustreError: 334561:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10686.945718] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10686.951612] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10686.958873] LustreError: Skipped 1 previous similar message [10686.970847] Lustre: Skipped 1 previous similar message [10686.972127] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10686.992416] Lustre: Skipped 1 previous similar message [10687.984357] LustreError: 334563:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10688.016583] LustreError: 334563:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10693.113244] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10693.120835] Lustre: Skipped 3 previous similar messages [10698.216522] LustreError: 334562:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10698.236578] LustreError: 334562:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10701.280145] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10701.567642] Lustre: server umount lustre-OST0000 complete [10707.937322] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10707.941823] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10707.949366] Lustre: Skipped 1 previous similar message [10707.953469] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10707.957196] Lustre: Skipped 2 previous similar messages [10712.105353] Lustre: server umount lustre-MDT0000 complete [10716.053585] LustreError: 332199:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788635083 with bad export cookie 13440376473986284998 [10716.056879] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10716.070729] LustreError: 332199:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [10716.460188] Lustre: server umount lustre-MDT0001 complete [10727.502938] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [10730.193369] Key type lgssc unregistered [10730.459383] LNet: 337595:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10730.468568] LNetError: 337595:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10730.494198] LNet: Removed LNI 192.168.202.127@tcp [10731.235760] Key type .llcrypt unregistered [10731.237743] Key type ._llcrypt unregistered [10808.604195] Key type ._llcrypt registered [10808.606943] Key type .llcrypt registered [10808.712923] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_hostid [10826.766925] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [10828.366313] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10828.422355] alg: No test for adler32 (adler32-zlib) [10829.598760] Lustre: Lustre: Build Version: 2.17.58_2_g69c2166 [10829.935995] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [10831.736169] Key type lgssc registered [10833.066729] Lustre: Echo OBD driver; http://www.lustre.org/ [10872.898813] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10872.920742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10874.310983] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10874.380865] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10874.510070] Lustre: lustre-MDT0000: new disk, initializing [10874.638529] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10874.656769] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10879.975896] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [10891.057566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10891.230983] Lustre: 342530:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10891.293711] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10891.297392] Lustre: Skipped 1 previous similar message [10891.402533] Lustre: lustre-MDT0001: new disk, initializing [10891.538137] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10891.598598] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10891.607837] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10896.092743] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [10906.008832] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10912.518989] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10920.069646] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10920.479403] Lustre: lustre-OST0000: new disk, initializing [10920.486710] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10920.498920] Lustre: 344498:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10920.583283] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10927.203538] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10927.220090] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10927.312574] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10928.078443] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [10937.408288] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10943.647273] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10943.909165] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10950.195231] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10950.384880] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10958.318277] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10958.335966] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10958.361990] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10958.370147] Lustre: Skipped 1 previous similar message [10959.525988] Lustre: server umount lustre-OST0000 complete [10968.554350] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10968.566712] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10968.569634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10968.576626] Lustre: Skipped 2 previous similar messages [10970.274341] Lustre: server umount lustre-MDT0000 complete [10974.708478] LustreError: 342523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788635341 with bad export cookie 16154124699996836936 [10974.708853] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10974.715699] LustreError: 342523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10975.009873] Lustre: server umount lustre-MDT0001 complete [10985.034915] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 15:09:10 (1788635350) [10996.501349] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 15:09:21 (1788635361) [11006.661256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11007.313576] LustreError: 347258:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11007.328041] LustreError: 347258:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11007.427853] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11011.175682] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [11012.593250] LustreError: 347259:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11017.708145] LustreError: 347258:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11021.564340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11022.094822] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11028.586177] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [11037.152576] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11045.595220] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11054.426837] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11054.760587] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11062.923600] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [11072.132681] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11084.635793] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [11086.571458] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [11088.408961] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [11090.548118] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [11092.907758] Lustre: Failing over lustre-MDT0000 [11092.962298] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11092.983065] Lustre: Skipped 1 previous similar message [11092.988946] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11092.993282] Lustre: Skipped 1 previous similar message [11093.354678] Lustre: server umount lustre-MDT0000 complete [11093.497030] LustreError: 348994:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11098.081146] LustreError: 347277:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11098.094691] LustreError: 347277:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11108.328831] LustreError: 347258:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11108.348770] LustreError: 347258:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [11113.204285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11113.335396] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11113.442835] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [11113.451206] Lustre: Skipped 4 previous similar messages [11113.681523] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11113.760512] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11114.946903] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11118.902485] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [11119.089876] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11119.175339] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [11119.235335] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11129.823885] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11131.596215] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11135.209406] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [11137.496047] Lustre: Failing over lustre-MDT0000 [11137.888699] Lustre: server umount lustre-MDT0000 complete [11139.556258] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11139.568709] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11139.577046] Lustre: Skipped 1 previous similar message [11139.585108] LustreError: 347258:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11139.599893] LustreError: 347258:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11155.872101] Lustre: 340755:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788635506/real 1788635506] req@00000000497324a9 x1875519939377152/t0(0) o400->MGC192.168.202.127@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788635522 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11155.902573] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11156.359490] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11165.153197] LustreError: 340751:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000049c0557b x1875519939382528/t0(0) o250->MGC192.168.202.127@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11165.427828] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11165.469624] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11167.185361] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11170.816685] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11170.837868] Lustre: Skipped 2 previous similar messages [11170.891436] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [11170.927132] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11171.248376] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [11182.600187] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11184.654661] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11196.389274] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11196.407326] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11196.414109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11196.440621] Lustre: Skipped 2 previous similar messages [11198.512478] Lustre: server umount lustre-MDT0000 complete [11201.506963] LustreError: 347252:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11201.518300] LustreError: 347252:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 26 previous similar messages [11202.723467] LustreError: 347993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788635569 with bad export cookie 16154124699996839680 [11202.726918] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11202.730865] LustreError: 347993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11203.046751] Lustre: server umount lustre-MDT0001 complete [11217.404108] Lustre: server umount lustre-OST0000 complete [11236.382560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11236.873512] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11241.867752] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [11251.521701] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11257.486362] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [11265.823486] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11273.081323] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11280.117178] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11281.629750] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11287.102298] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [11294.692448] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11311.974752] Lustre: server umount lustre-OST0000 complete [11317.737140] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11317.744285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11317.746871] Lustre: Skipped 2 previous similar messages [11317.762681] Lustre: Skipped 3 previous similar messages [11328.126102] Lustre: server umount lustre-MDT0000 complete [11328.993156] Lustre: 340754:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788635679/real 1788635679] req@00000000c072493e x1875519939432832/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788635695 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11329.001863] LustreError: 353624:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11329.032211] Lustre: 340754:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [11329.066703] LustreError: 353624:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11331.701411] LustreError: 353605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788635698 with bad export cookie 16154124699996840394 [11331.711696] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11332.264650] Lustre: server umount lustre-MDT0001 complete [11345.845674] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [11349.178296] Key type lgssc unregistered [11349.564769] LNet: 357182:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11349.568303] LNetError: 357182:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11349.579441] LNet: Removed LNI 192.168.202.127@tcp [11350.669120] Key type .llcrypt unregistered [11350.670852] Key type ._llcrypt unregistered [11381.604038] Key type ._llcrypt registered [11381.606278] Key type .llcrypt registered [11381.748056] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [11383.020227] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11383.238094] alg: No test for adler32 (adler32-zlib) [11384.292862] Lustre: Lustre: Build Version: 2.17.58_2_g69c2166 [11384.676747] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [11386.401749] Key type lgssc registered [11387.658488] Lustre: Echo OBD driver; http://www.lustre.org/ [11396.275375] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11396.294553] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11397.851207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11403.024481] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [11407.332506] LustreError: 358698:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11407.348764] LustreError: 358698:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11411.885994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11412.208729] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11417.148648] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [11425.897756] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11431.721848] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11439.569527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11440.110571] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11442.178835] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11447.325931] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [11455.442167] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11467.752465] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11467.755216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11467.766061] Lustre: Skipped 1 previous similar message [11467.775962] Lustre: Skipped 2 previous similar messages [11468.622556] Lustre: server umount lustre-MDT0000 complete [11471.997383] LustreError: 358678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788635839 with bad export cookie 14931674186696972089 [11472.006725] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11472.686933] Lustre: server umount lustre-MDT0001 complete [11487.567222] Lustre: server umount lustre-OST0000 complete [11502.134234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11502.675554] LustreError: 362408:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11502.770035] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11508.195140] LustreError: 362409:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11508.207955] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [11512.298463] LustreError: 362408:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11517.412399] LustreError: 362409:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11518.093377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11518.488507] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11523.153850] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [11533.832604] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11542.472927] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11552.027206] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11552.319056] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11558.271429] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [11566.769339] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11572.803405] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11593.188530] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11593.197457] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11593.204545] Lustre: Skipped 3 previous similar messages [11593.217579] Lustre: Skipped 2 previous similar messages [11597.535243] Lustre: server umount lustre-MDT0000 complete [11598.313197] LustreError: 362402:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11598.329057] LustreError: 362402:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11602.567287] LustreError: 364252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788635969 with bad export cookie 14931674186696973202 [11602.570661] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11602.579472] LustreError: 364252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11602.982836] Lustre: server umount lustre-MDT0001 complete [11617.376539] Lustre: server umount lustre-OST0000 complete [11635.242605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11635.813069] LustreError: 366348:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11635.895517] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11640.514112] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [11650.647381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11657.281240] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [11667.146289] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11673.896596] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11680.278498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11680.611814] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11680.615963] Lustre: Skipped 1 previous similar message [11681.677240] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11687.592803] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [11697.654454] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11706.342634] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11706.346111] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11706.356346] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11711.462125] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11711.467458] Lustre: Skipped 1 previous similar message [11716.581068] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11716.596564] Lustre: Skipped 2 previous similar messages [11717.088295] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11717.269289] Lustre: server umount lustre-OST0000 complete [11722.722507] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11722.746460] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11722.771134] Lustre: Skipped 1 previous similar message [11726.818322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11726.825241] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11726.832306] Lustre: Skipped 1 previous similar message [11728.304603] Lustre: server umount lustre-MDT0000 complete [11731.937770] LustreError: 366348:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11731.977834] LustreError: 366348:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [11732.736860] LustreError: 366328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788636099 with bad export cookie 14931674186696974224 [11732.746083] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11733.251762] Lustre: server umount lustre-MDT0001 complete [11746.814065] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [11749.721735] Key type lgssc unregistered [11750.002837] LNet: 369883:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11750.007489] LNetError: 369883:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11750.019857] LNet: Removed LNI 192.168.202.127@tcp [11750.926805] Key type .llcrypt unregistered [11750.928836] Key type ._llcrypt unregistered [11781.981848] Key type ._llcrypt registered [11781.987133] Key type .llcrypt registered [11782.156539] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [11783.324508] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11783.361109] alg: No test for adler32 (adler32-zlib) [11784.383757] Lustre: Lustre: Build Version: 2.17.58_2_g69c2166 [11784.585647] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [11786.224241] Key type lgssc registered [11787.799883] Lustre: Echo OBD driver; http://www.lustre.org/ [11795.574305] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11795.587033] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11795.593448] LustreError: 371334:0:(osd_handler.c:8546:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11795.607857] LustreError: 371334:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11795.618183] LustreError: 371334:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11795.623641] LustreError: 371334:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11795.635605] LustreError: 371334:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11804.598839] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11804.602223] LustreError: 371731:0:(osd_handler.c:8546:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11804.621123] LustreError: 371731:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11804.627408] LustreError: 371731:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11804.634474] LustreError: 371731:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11804.644936] LustreError: 371731:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11829.322693] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [11832.430517] Key type lgssc unregistered [11832.614440] LNet: 372734:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11832.621057] LNetError: 372734:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11832.634622] LNet: Removed LNI 192.168.202.127@tcp [11833.132422] Key type .llcrypt unregistered [11833.134246] Key type ._llcrypt unregistered [11852.703518] Key type ._llcrypt registered [11852.706232] Key type .llcrypt registered [11852.839442] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_hostid [11868.573308] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [11869.728674] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11869.901303] alg: No test for adler32 (adler32-zlib) [11871.093398] Lustre: Lustre: Build Version: 2.17.58_2_g69c2166 [11871.337883] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [11872.976630] Key type lgssc registered [11874.080046] Lustre: Echo OBD driver; http://www.lustre.org/ [11914.775416] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11914.823179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11916.052532] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11916.088520] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11916.145291] Lustre: lustre-MDT0000: new disk, initializing [11916.215485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11916.242205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11921.172756] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [11932.783462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11933.019973] Lustre: 375889:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11933.090692] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11933.094903] Lustre: Skipped 1 previous similar message [11933.213552] Lustre: lustre-MDT0001: new disk, initializing [11933.359768] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11933.408903] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11933.419260] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11938.555836] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [11947.797284] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11953.895132] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11961.445018] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11961.677860] Lustre: lustre-OST0000: new disk, initializing [11961.680473] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11961.686778] Lustre: 377858:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11961.767857] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11969.146400] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11969.189655] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11969.488542] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11969.561256] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [11980.160907] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11985.350122] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11985.646486] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11990.625955] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11990.947194] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11995.117921] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11995.124725] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11995.131795] Lustre: Skipped 1 previous similar message [11995.143753] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11995.146168] LustreError: Skipped 1 previous similar message [11999.637241] Lustre: server umount lustre-OST0000 complete [12005.352115] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12005.354386] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12005.377388] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12005.383472] Lustre: Skipped 1 previous similar message [12005.398772] Lustre: Skipped 2 previous similar messages [12010.465484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12010.475857] Lustre: Skipped 1 previous similar message [12010.574689] Lustre: server umount lustre-MDT0000 complete [12014.481804] LustreError: 375881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788636381 with bad export cookie 12820133883174942294 [12014.493098] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12015.041426] Lustre: server umount lustre-MDT0001 complete [12023.833857] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [12025.513857] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 15:26:31 (1788636391) [12031.674627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12032.121913] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12035.951757] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [12037.614280] LustreError: 380316:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12037.633981] LustreError: 380316:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12042.723563] LustreError: 380315:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12043.712871] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12043.961201] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12049.016492] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [12056.940341] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12063.102538] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12070.975275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12071.232700] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12078.099403] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [12086.971516] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12094.559628] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12094.674532] Lustre: lustre-OST0001: new disk, initializing [12094.677967] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [12094.684031] Lustre: 383154:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12094.774367] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12101.997233] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [12104.772215] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12104.781378] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12104.839112] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12111.965961] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12119.013434] Lustre: lustre-OST0001: Not available for connect from 192.168.202.27@tcp (stopping) [12120.547660] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12120.558280] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12121.859808] Lustre: server umount lustre-OST0001 complete [12124.102519] LustreError: 382137:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12130.798406] LustreError: 382137:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12130.826752] LustreError: 382137:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12135.925120] LustreError: 384079:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12135.948193] LustreError: 384079:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12138.981185] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12138.984410] LustreError: Skipped 1 previous similar message [12138.987055] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12138.991766] Lustre: Skipped 1 previous similar message [12138.994541] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12138.999310] Lustre: Skipped 2 previous similar messages [12146.163645] LustreError: 382138:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12146.184852] LustreError: 382138:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12151.274880] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12151.284625] Lustre: Skipped 6 previous similar messages [12153.312783] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12153.598779] Lustre: server umount lustre-OST0000 complete [12156.897549] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12156.908511] LustreError: Skipped 1 previous similar message [12156.911443] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12156.919374] Lustre: Skipped 1 previous similar message [12171.232091] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12171.465458] Lustre: server umount lustre-MDT0000 complete [12171.746044] LustreError: 380311:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12171.760832] LustreError: 380311:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12175.420708] LustreError: 380297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788636542 with bad export cookie 12820133883174943176 [12175.425402] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12175.435615] LustreError: 380297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [12182.113040] Lustre: server umount lustre-MDT0001 complete [12198.686469] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [12201.623193] Key type lgssc unregistered [12202.000994] LNet: 385144:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12202.009612] LNetError: 385144:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12202.021732] LNet: Removed LNI 192.168.202.127@tcp [12203.010812] Key type .llcrypt unregistered [12203.015467] Key type ._llcrypt unregistered [12261.974297] Key type ._llcrypt registered [12261.977823] Key type .llcrypt registered [12262.101630] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [12263.649674] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12263.671545] alg: No test for adler32 (adler32-zlib) [12264.814220] Lustre: Lustre: Build Version: 2.17.58_2_g69c2166 [12265.071807] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [12266.744339] Key type lgssc registered [12268.092666] Lustre: Echo OBD driver; http://www.lustre.org/ [12276.556799] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12276.603709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12277.920753] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12277.944621] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12278.137732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12283.062204] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [12292.636943] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12292.773741] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12292.803342] Lustre: 387962:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12293.204875] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12298.324902] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [12306.716360] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12313.213425] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12317.157647] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12317.165064] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12317.171591] Lustre: Skipped 1 previous similar message [12317.188791] Lustre: Skipped 1 previous similar message [12322.380521] Lustre: server umount lustre-MDT0000 complete [12327.157369] LustreError: 387955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788636694 with bad export cookie 7414377250124403121 [12327.161115] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12327.164533] LustreError: 387955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12327.576847] Lustre: server umount lustre-MDT0001 complete [12341.400526] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 15:31:46 (1788636706) [12350.647233] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12351.085098] LustreError: 390476:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12351.108180] LustreError: 390476:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12351.151856] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12356.586444] LustreError: 390477:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12356.796145] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [12361.701167] LustreError: 390476:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12366.818149] LustreError: 390477:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12366.882602] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12367.276359] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12372.249453] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [12382.249394] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12389.444933] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12397.682084] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12397.897411] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [12398.064857] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12406.901282] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [12416.162718] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12424.881286] Lustre: Permanently deactivating lustre-OST0000 [12424.905777] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [12434.016388] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12435.570854] LustreError: 393465:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [12435.578349] LustreError: 393465:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12435.584569] LustreError: 393465:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12438.043747] LustreError: 393513:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12438.054909] LustreError: 393513:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12446.354430] Lustre: server umount lustre-OST0000 complete [12449.880227] LustreError: 393764:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12449.887217] LustreError: 393764:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12454.370055] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12454.380104] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12454.400510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12456.929467] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12456.934516] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12461.369296] Lustre: server umount lustre-MDT0000 complete [12462.049828] LustreError: 390477:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12462.071803] LustreError: 390477:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12466.518326] LustreError: 390457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788636833 with bad export cookie 7414377250124403702 [12466.530195] LustreError: 390457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12466.531558] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12477.030524] Lustre: server umount lustre-MDT0001 complete [12488.048848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12495.913418] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [12499.691065] LustreError: 394926:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12506.252976] Lustre: 395266:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.127@tcp [12508.142940] Lustre: server umount MGS complete [12523.834503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12524.235586] LustreError: 395990:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12524.247307] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12524.258385] Lustre: Skipped 1 previous similar message [12524.299275] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12529.697839] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [12539.703236] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12539.882052] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [12539.897990] Lustre: Skipped 1 previous similar message [12540.017658] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12545.034404] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [12553.369275] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12559.787316] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12562.064454] Lustre: Permanently reactivating lustre-OST0000 [12562.079990] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12562.082406] Lustre: Skipped 2 previous similar messages [12588.002425] LustreError: 397722:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12595.865127] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12596.287147] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12596.290113] Lustre: Skipped 1 previous similar message [12603.828408] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [12613.014222] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12622.819618] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12622.827107] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12622.845645] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12624.353268] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12625.556443] Lustre: server umount lustre-OST0000 complete [12632.547874] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12632.554409] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12632.567357] Lustre: Skipped 1 previous similar message [12632.573012] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12632.580376] Lustre: Skipped 2 previous similar messages [12644.320541] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12644.630356] Lustre: server umount lustre-MDT0000 complete [12644.851596] LustreError: 395990:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12644.877555] LustreError: 395990:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12648.876432] LustreError: 395971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788637015 with bad export cookie 7414377250124404752 [12648.880560] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12648.885198] LustreError: 395971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12649.475376] Lustre: server umount lustre-MDT0001 complete [12660.897921] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [12663.951571] Key type lgssc unregistered [12664.298311] LNet: 400260:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12664.305650] LNetError: 400260:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12664.330862] LNet: Removed LNI 192.168.202.127@tcp [12665.365061] Key type .llcrypt unregistered [12665.366661] Key type ._llcrypt unregistered [12686.649635] Key type ._llcrypt registered [12686.651845] Key type .llcrypt registered [12686.792837] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_hostid [12703.070656] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [12704.096238] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12704.352682] alg: No test for adler32 (adler32-zlib) [12705.483256] Lustre: Lustre: Build Version: 2.17.58_2_g69c2166 [12705.765507] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [12707.480170] Key type lgssc registered [12708.576373] Lustre: Echo OBD driver; http://www.lustre.org/ [12751.314904] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12751.377922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12752.801303] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12752.860984] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12752.994824] Lustre: lustre-MDT0000: new disk, initializing [12753.119768] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12753.140669] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12756.948342] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [12767.862699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12767.993946] Lustre: 403416:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12768.044156] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12768.047934] Lustre: Skipped 1 previous similar message [12768.164459] Lustre: lustre-MDT0001: new disk, initializing [12768.321181] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12768.369484] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12768.395414] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12772.571886] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [12783.411826] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12789.858540] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12797.003986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12797.467362] Lustre: lustre-OST0000: new disk, initializing [12797.475554] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12797.480801] Lustre: 405385:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12797.617651] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12804.273288] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [12807.289227] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12807.303759] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12807.411815] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12815.705786] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12821.401464] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12821.691416] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12826.647435] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12826.817918] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12833.254755] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12833.260666] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12833.278675] LustreError: Skipped 1 previous similar message [12833.284638] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12833.312940] Lustre: Skipped 1 previous similar message [12833.343665] Lustre: Skipped 1 previous similar message [12835.500296] Lustre: server umount lustre-OST0000 complete [12839.911759] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12839.915233] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12839.927881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12843.491470] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12843.493172] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12845.705646] Lustre: server umount lustre-MDT0000 complete [12849.666700] LustreError: 403408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788637216 with bad export cookie 4091496484920807639 [12849.670038] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12849.677666] LustreError: 403408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12850.152648] Lustre: server umount lustre-MDT0001 complete [12861.539510] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12864.131570] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 15:40:29 (1788637229) [12872.739445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12873.420511] LustreError: 407847:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12873.454322] LustreError: 407847:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12873.551499] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12878.818338] LustreError: 407848:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12879.011595] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [12883.941666] LustreError: 407847:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12888.885505] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12889.075439] LustreError: 407848:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12889.495188] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12895.638874] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [12905.531185] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12913.877233] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12923.067573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12923.693077] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12931.386930] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [12940.710762] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12943.079764] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12945.910895] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12953.585050] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12953.588391] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12953.627000] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12953.633266] Lustre: Skipped 1 previous similar message [12958.793605] Lustre: server umount lustre-OST0000 complete [12963.811971] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12963.816154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12963.827660] Lustre: Skipped 2 previous similar messages [12970.102606] Lustre: server umount lustre-MDT0000 complete [12973.029603] LustreError: 407847:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12973.046143] LustreError: 407847:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12975.088204] LustreError: 407828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788637342 with bad export cookie 4091496484920808521 [12975.102852] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12975.112550] LustreError: 407828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12976.005180] Lustre: server umount lustre-MDT0001 complete [12990.281458] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [12994.427917] Key type lgssc unregistered [12994.749341] LNet: 411571:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12994.761929] LNetError: 411571:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12994.788338] LNet: Removed LNI 192.168.202.127@tcp [12995.920370] Key type .llcrypt unregistered [12995.922804] Key type ._llcrypt unregistered [13025.146180] Key type ._llcrypt registered [13025.147599] Key type .llcrypt registered [13025.284291] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [13025.597401] Key type .llcrypt unregistered [13025.599609] Key type ._llcrypt unregistered [13045.715794] Key type ._llcrypt registered [13045.719389] Key type .llcrypt registered [13045.888671] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [13047.245119] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13047.256961] alg: No test for adler32 (adler32-zlib) [13048.452123] Lustre: Lustre: Build Version: 2.17.58_2_g69c2166 [13048.713917] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [13050.393114] Key type lgssc registered [13051.718490] Lustre: Echo OBD driver; http://www.lustre.org/ [13061.409676] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13061.439886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13062.978206] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13068.321606] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [13072.359548] LustreError: 413781:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13072.372792] LustreError: 413781:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13077.482231] LustreError: 413782:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13078.422157] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13078.892103] LustreError: 413781:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13078.927432] LustreError: 413781:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13078.947077] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13084.142550] LustreError: 413782:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13084.170966] LustreError: 413782:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13085.967417] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [13087.214494] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [13092.329049] LustreError: 413781:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13092.356285] LustreError: 413781:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [13094.587678] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13101.953308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13102.271771] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13106.910294] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [13117.931774] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13117.941346] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13117.971332] LustreError: Skipped 1 previous similar message [13117.972623] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13123.049574] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13123.058066] Lustre: Skipped 1 previous similar message [13128.165667] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13128.172976] Lustre: Skipped 2 previous similar messages [13130.208469] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13130.494770] Lustre: server umount lustre-OST0000 complete [13138.404165] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13138.405061] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13138.408779] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13138.441961] Lustre: Skipped 2 previous similar messages [13140.877646] Lustre: server umount lustre-MDT0000 complete [13143.521815] LustreError: 415517:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13143.535859] LustreError: 415517:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [13144.787614] LustreError: 413762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788637511 with bad export cookie 15417777271943475793 [13144.788732] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13144.804791] LustreError: 413762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13145.402725] Lustre: server umount lustre-MDT0001 complete [13159.378703] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [13162.254879] Key type lgssc unregistered [13162.543051] LNet: 416931:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13162.551241] LNetError: 416931:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13162.577083] LNet: Removed LNI 192.168.202.127@tcp [13163.512594] Key type .llcrypt unregistered [13163.514289] Key type ._llcrypt unregistered [13191.304867] Key type ._llcrypt registered [13191.306974] Key type .llcrypt registered [13191.444524] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [13192.544972] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13192.632219] alg: No test for adler32 (adler32-zlib) [13193.762151] Lustre: Lustre: Build Version: 2.17.58_2_g69c2166 [13193.983464] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [13195.672197] Key type lgssc registered [13197.451116] Lustre: Echo OBD driver; http://www.lustre.org/ [13205.357587] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13205.373886] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13220.897148] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [13226.016935] LustreError: 418317:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.127@tcp: failed processing log, type 4: rc = -110 [13252.704219] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13259.657895] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [13267.829837] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13274.953457] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13275.534962] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13275.584579] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [13280.113873] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [13280.743923] LustreError: 418343:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13280.769426] LustreError: 418343:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13285.878088] LustreError: 418343:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13288.931537] LustreError: 419355:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13288.952811] LustreError: 419355:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13289.170083] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13289.567604] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13293.812361] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [13300.441324] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13305.884742] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13315.042826] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13315.048034] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13315.067364] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13315.074215] Lustre: Skipped 1 previous similar message [13315.086035] Lustre: Skipped 1 previous similar message [13315.648253] Lustre: server umount lustre-OST0000 complete [13320.162161] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13320.171284] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13320.174620] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13320.194647] Lustre: Skipped 1 previous similar message [13325.288245] LustreError: 419359:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13325.557322] Lustre: server umount lustre-MDT0000 complete [13329.415518] LustreError: 418323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788637696 with bad export cookie 1761728063472472882 [13329.421625] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13329.425512] LustreError: 418323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13329.830943] Lustre: server umount lustre-MDT0001 complete [13342.308343] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [13345.624788] Key type lgssc unregistered [13346.070588] LNet: 421887:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13346.078733] LNetError: 421887:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13347.114265] LNet: Removed LNI 192.168.202.127@tcp [13347.984794] Key type .llcrypt unregistered [13347.987589] Key type ._llcrypt unregistered [13373.021491] Key type ._llcrypt registered [13373.027197] Key type .llcrypt registered [13373.195761] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [13374.615548] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13374.708533] alg: No test for adler32 (adler32-zlib) [13375.722623] Lustre: Lustre: Build Version: 2.17.58_2_g69c2166 [13375.951397] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [13377.624166] Key type lgssc registered [13378.740450] Lustre: Echo OBD driver; http://www.lustre.org/ [13386.026416] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13386.045595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13387.428448] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13391.440886] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [13397.987879] LustreError: 423303:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13398.007299] LustreError: 423303:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13399.265553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13399.704039] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13405.013628] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [13414.296970] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13421.894875] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13429.663986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13430.076961] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13433.162091] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [13437.262996] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [13445.068301] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13449.712846] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13449.726734] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13449.736708] Lustre: Skipped 1 previous similar message [13449.750687] Lustre: Skipped 2 previous similar messages [13451.745898] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.27@tcp (stopping) [13454.828310] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13454.842895] Lustre: Skipped 1 previous similar message [13454.973410] Lustre: server umount lustre-MDT0000 complete [13456.871453] LustreError: 423299:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13458.055849] LustreError: 423283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.27@tcp arrived at 1788637825 with bad export cookie 9041851224937651121 [13458.078255] LustreError: 423283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13459.938099] LustreError: 423299:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13459.960623] LustreError: 423299:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13460.488957] LustreError: 423282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788637827 with bad export cookie 9041851224937650225 [13460.504339] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13460.505281] LustreError: 423282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13465.058873] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13465.060938] LustreError: 424030:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13465.062572] Lustre: Skipped 1 previous similar message [13465.069638] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13465.102635] LustreError: 424030:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13465.106499] Lustre: Skipped 1 previous similar message [13466.728377] Lustre: server umount lustre-OST0000 complete [13470.180959] LustreError: 423302:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13470.223740] LustreError: 423302:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13475.536343] Lustre: server umount lustre-MDT0001 complete [13488.077624] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [13491.409903] Key type lgssc unregistered [13491.812203] LNet: 426988:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13491.820781] LNetError: 426988:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13491.864841] LNet: Removed LNI 192.168.202.127@tcp [13493.080772] Key type .llcrypt unregistered [13493.083713] Key type ._llcrypt unregistered [13519.772252] Key type ._llcrypt registered [13519.778329] Key type .llcrypt registered [13519.893810] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [13521.411596] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13521.466624] alg: No test for adler32 (adler32-zlib) [13522.563902] Lustre: Lustre: Build Version: 2.17.58_2_g69c2166 [13522.829871] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [13524.544175] Key type lgssc registered [13525.954119] Lustre: Echo OBD driver; http://www.lustre.org/ [13534.109232] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13534.133160] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13535.610075] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13540.668258] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [13545.956273] LustreError: 428403:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13545.973882] LustreError: 428403:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13549.539661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13550.187951] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13555.208885] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [13563.375714] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13570.334580] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13577.983144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13578.446141] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13580.478336] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13585.835967] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [13595.862213] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13601.255875] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13601.272110] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13601.275179] Lustre: Skipped 1 previous similar message [13601.283650] Lustre: Skipped 1 previous similar message [13602.804070] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.27@tcp (stopping) [13603.680842] Lustre: server umount lustre-MDT0001 complete [13606.382167] LustreError: 428403:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13606.420812] LustreError: 428403:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13611.494374] LustreError: 429132:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13611.524371] LustreError: 429132:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13616.609970] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13616.614178] LustreError: 428404:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13616.629772] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13616.639820] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13620.819033] Lustre: server umount lustre-OST0000 complete [13621.732482] LustreError: 429132:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13626.058913] Lustre: server umount lustre-MDT0000 complete [13641.572894] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [13644.920235] Key type lgssc unregistered [13645.281048] LNet: 432088:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13645.298235] LNetError: 432088:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13645.329050] LNet: Removed LNI 192.168.202.127@tcp [13646.232752] Key type .llcrypt unregistered [13646.235088] Key type ._llcrypt unregistered [13674.650770] Key type ._llcrypt registered [13674.653783] Key type .llcrypt registered [13674.759309] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_hostid [13692.823203] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [13694.040870] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13694.194563] alg: No test for adler32 (adler32-zlib) [13695.288380] Lustre: Lustre: Build Version: 2.17.58_2_g69c2166 [13695.501283] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [13697.160189] Key type lgssc registered [13698.161135] Lustre: Echo OBD driver; http://www.lustre.org/ [13744.980972] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [13764.237728] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13764.263190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13765.595930] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13765.624946] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13765.683373] Lustre: lustre-MDT0000: new disk, initializing [13765.767399] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13765.782617] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13770.870187] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [13787.679267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13787.952467] Lustre: lustre-OST0000: new disk, initializing [13787.956303] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13787.960981] Lustre: Skipped 1 previous similar message [13787.967079] Lustre: 437121:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13788.045858] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13794.386617] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [13794.435505] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13794.442506] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13794.470131] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13806.128850] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13820.925631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13821.070902] Lustre: 436105:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13821.129812] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13821.219418] Lustre: lustre-MDT0001: new disk, initializing [13821.344274] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13821.392412] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13821.402305] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13825.393513] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [13835.230735] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13835.466565] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13840.022860] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13840.277151] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13846.501161] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13846.509487] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13846.523537] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13846.534741] Lustre: Skipped 1 previous similar message [13848.066834] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.27@tcp (stopping) [13851.522869] Lustre: server umount lustre-MDT0001 complete [13856.745797] LustreError: 436111:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13856.770667] LustreError: 436111:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [13861.878748] LustreError: 436114:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13861.905328] LustreError: 436114:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13866.981313] LustreError: 436111:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13866.994135] LustreError: 436111:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13871.588291] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13871.611316] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13871.635246] Lustre: Skipped 1 previous similar message [13871.648234] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13872.102825] LustreError: 436114:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13875.180371] Lustre: server umount lustre-OST0000 complete [13877.226229] LustreError: 436115:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13885.418204] Lustre: server umount lustre-MDT0000 complete [13899.699826] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [13902.378128] Key type lgssc unregistered [13902.655557] LNet: 440866:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13902.659424] LNetError: 440866:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13902.673870] LNet: Removed LNI 192.168.202.127@tcp [13903.569740] Key type .llcrypt unregistered [13903.573874] Key type ._llcrypt unregistered [13952.260899] Key type ._llcrypt registered [13952.265476] Key type .llcrypt registered [13952.407888] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [13952.855294] Key type .llcrypt unregistered [13952.860767] Key type ._llcrypt unregistered [13972.364681] Key type ._llcrypt registered [13972.366423] Key type .llcrypt registered [13972.516756] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [13973.826268] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13974.066576] alg: No test for adler32 (adler32-zlib) [13975.339633] Lustre: Lustre: Build Version: 2.17.58_2_g69c2166 [13975.752230] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [13977.432513] Key type lgssc registered [13978.540677] Lustre: Echo OBD driver; http://www.lustre.org/ [14008.934377] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14008.999934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14010.330466] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14010.366573] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14010.492532] Lustre: lustre-MDT0000: new disk, initializing [14010.591826] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14010.632735] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14015.683968] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [14028.509443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14028.621715] Lustre: 444237:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14028.688597] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14028.695378] Lustre: Skipped 1 previous similar message [14028.781514] Lustre: lustre-MDT0001: new disk, initializing [14028.878207] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14028.912766] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14028.925180] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14033.993191] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [14043.990940] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14050.356915] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14058.295872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14058.643395] Lustre: lustre-OST0000: new disk, initializing [14058.647960] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14058.653736] Lustre: 446206:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14058.775849] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14066.127198] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [14067.345036] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14067.365983] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14067.529664] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14076.564463] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14088.163392] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14088.165685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14088.180662] Lustre: Skipped 1 previous similar message [14088.205418] Lustre: Skipped 2 previous similar messages [14091.538565] Lustre: server umount lustre-MDT0000 complete [14094.952670] LustreError: 444231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788638461 with bad export cookie 10004670067502335154 [14094.956213] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14094.966675] LustreError: 444231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14095.231932] Lustre: server umount lustre-MDT0001 complete [14109.522448] Lustre: server umount lustre-OST0000 complete [14122.851810] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 16:01:28 (1788638488) [14130.969444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14131.331396] LustreError: 448245:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14131.343135] LustreError: 448245:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [14131.371904] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14136.806543] LustreError: 448246:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14137.715488] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [14141.934491] LustreError: 448245:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14147.066800] LustreError: 448246:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14147.623779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14148.107215] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14153.965348] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [14163.588945] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14170.001980] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14191.027952] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14191.227774] Lustre: Found index 0 for lustre-OST0000, updating log [14191.407500] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14199.343787] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [14202.690487] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [14209.592787] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14217.263650] Lustre: server umount lustre-OST0000 complete [14233.593604] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14233.601742] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14233.611256] Lustre: Skipped 2 previous similar messages [14233.631902] Lustre: Skipped 1 previous similar message [14234.337208] Lustre: 442769:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788638585/real 1788638585] req@000000009e30aa12 x1875523238178048/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788638601 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14235.424184] Lustre: 442770:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788638586/real 1788638586] req@000000002cc34d30 x1875523238178432/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788638602 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14239.520194] Lustre: 442769:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788638590/real 1788638590] req@0000000043439332 x1875523238179456/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788638606 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14239.724138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14239.729113] Lustre: Skipped 3 previous similar messages [14243.919737] Lustre: server umount lustre-MDT0000 complete [14244.576160] Lustre: 442770:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788638595/real 1788638595] req@00000000d0471b67 x1875523238180608/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788638611 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14244.839478] LustreError: 448245:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14244.858985] LustreError: 448245:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14247.877381] LustreError: 448226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788638614 with bad export cookie 10004670067502336547 [14247.880179] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14247.891960] LustreError: 448226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14248.244612] Lustre: server umount lustre-MDT0001 complete [14252.258156] LNet: 1 local NIs in recovery (showing 1): 192.168.202.127@tcp [14258.546906] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 16:03:44 (1788638624) [14884.524892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14885.003460] LustreError: 454660:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14885.025917] LustreError: 454660:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [14885.075318] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14889.432913] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [14890.481149] LustreError: 454661:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14895.590699] LustreError: 454660:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14899.233269] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14900.142937] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14907.087155] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [14918.574554] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14926.204965] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14942.625543] LNet: 1 local NIs in recovery (showing 1): 192.168.202.127@tcp [14948.222523] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14948.628259] Lustre: Found index 0 for lustre-OST0000, updating log [14948.946314] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14957.086345] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [14966.031878] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14968.659671] Lustre: server umount lustre-OST0000 complete [14974.446684] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14974.448032] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14974.469801] Lustre: Skipped 2 previous similar messages [14974.492397] Lustre: Skipped 1 previous similar message [14975.526104] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14975.541426] Lustre: Skipped 1 previous similar message [14976.554703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14978.595423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14978.623299] Lustre: Skipped 4 previous similar messages [14981.126728] Lustre: server umount lustre-MDT0000 complete [14981.665928] LustreError: 454660:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14981.704770] LustreError: 454660:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [14985.898530] LustreError: 454642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788639352 with bad export cookie 10004670067502337625 [14985.902473] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14985.905520] LustreError: 454642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14986.151251] Lustre: server umount lustre-MDT0001 complete [15002.081634] LNet: 1 local NIs in recovery (showing 1): 192.168.202.127@tcp [15017.846283] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_hostid [15029.683832] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [15071.519290] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 16:17:17 (1788639437) [15093.435053] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [15097.126944] Key type lgssc unregistered [15097.376179] LNet: 461271:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15097.391377] LNetError: 461271:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15210.025296] LNet: Removed LNI 192.168.202.127@tcp [15211.086386] Key type .llcrypt unregistered [15211.088301] Key type ._llcrypt unregistered [15229.620296] Key type ._llcrypt registered [15229.622262] Key type .llcrypt registered [15229.731162] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [15231.321343] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15231.475985] alg: No test for adler32 (adler32-zlib) [15232.843395] Lustre: Lustre: Build Version: 2.17.58_2_g69c2166 [15233.186288] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [15234.921207] Key type lgssc registered [15236.625250] Lustre: Echo OBD driver; http://www.lustre.org/ [15241.781978] LNet: Added LNI 192.168.202.127@tcp100 [8/256/0/180] [15242.920820] LNet: Added LNI 192.168.202.127@tcp101 [8/256/0/180] [15244.282309] LNet: Added LNI 192.168.202.127@tcp102 [8/256/0/180] [15245.153229] LNet: Added LNI 192.168.202.127@tcp103 [8/256/0/180] [15246.208042] LNet: Added LNI 192.168.202.127@tcp104 [8/256/0/180] [15247.157114] LNet: Added LNI 192.168.202.127@tcp105 [8/256/0/180] [15248.169508] LNet: Added LNI 192.168.202.127@tcp106 [8/256/0/180] [15249.050115] LNet: Added LNI 192.168.202.127@tcp107 [8/256/0/180] [15249.989437] LNet: Added LNI 192.168.202.127@tcp108 [8/256/0/180] [15250.990954] LNet: Added LNI 192.168.202.127@tcp109 [8/256/0/180] [15251.892429] LNet: Added LNI 192.168.202.127@tcp110 [8/256/0/180] [15252.669525] LNet: Added LNI 192.168.202.127@tcp111 [8/256/0/180] [15253.610101] LNet: Added LNI 192.168.202.127@tcp112 [8/256/0/180] [15254.499777] LNet: Added LNI 192.168.202.127@tcp113 [8/256/0/180] [15255.505924] LNet: Added LNI 192.168.202.127@tcp114 [8/256/0/180] [15256.596441] LNet: Added LNI 192.168.202.127@tcp115 [8/256/0/180] [15257.368844] LNet: Added LNI 192.168.202.127@tcp116 [8/256/0/180] [15258.273247] LNet: Added LNI 192.168.202.127@tcp117 [8/256/0/180] [15259.159914] LNet: Added LNI 192.168.202.127@tcp118 [8/256/0/180] [15260.321421] LNet: Added LNI 192.168.202.127@tcp119 [8/256/0/180] [15261.340925] LNet: Added LNI 192.168.202.127@tcp120 [8/256/0/180] [15262.611242] LNet: Added LNI 192.168.202.127@tcp121 [8/256/0/180] [15263.842497] LNet: Added LNI 192.168.202.127@tcp122 [8/256/0/180] [15265.073248] LNet: Added LNI 192.168.202.127@tcp123 [8/256/0/180] [15266.468761] LNet: Added LNI 192.168.202.127@tcp124 [8/256/0/180] [15267.686411] LNet: Added LNI 192.168.202.127@tcp125 [8/256/0/180] [15268.795193] LNet: Added LNI 192.168.202.127@tcp126 [8/256/0/180] [15270.146936] LNet: Added LNI 192.168.202.127@tcp127 [8/256/0/180] [15271.691183] LNet: Added LNI 192.168.202.127@tcp128 [8/256/0/180] [15273.271895] LNet: Added LNI 192.168.202.127@tcp129 [8/256/0/180] [15274.572635] LNet: Added LNI 192.168.202.127@tcp130 [8/256/0/180] [15276.062053] LNet: Added LNI 192.168.202.127@tcp131 [8/256/0/180] [15277.459949] LNet: Added LNI 192.168.202.127@tcp132 [8/256/0/180] [15278.700776] LNet: Added LNI 192.168.202.127@tcp133 [8/256/0/180] [15281.244782] LNet: Added LNI 192.168.202.127@tcp134 [8/256/0/180] [15282.368397] LNet: Added LNI 192.168.202.127@tcp135 [8/256/0/180] [15290.604153] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15290.633127] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15291.773162] Lustre: 464301:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.127@tcp: can't add NID 192.168.202.127@tcp130: rc = -75 [15291.870042] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15291.888347] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15291.958814] Lustre: lustre-MDT0000: new disk, initializing [15292.024958] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15292.040212] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15298.442582] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [15310.953527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15311.126735] Lustre: 464316:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15311.238585] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15311.251393] Lustre: Skipped 1 previous similar message [15311.421497] Lustre: lustre-MDT0001: new disk, initializing [15311.513286] Lustre: 465114:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.127@tcp: can't add NID 192.168.202.127@tcp130: rc = -75 [15311.540652] Lustre: 465114:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15311.801704] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15311.901221] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15311.930536] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15319.568717] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [15320.739174] Lustre: 465401:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.127@tcp: can't add NID 192.168.202.127@tcp130: rc = -75 [15320.756822] Lustre: 465401:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15335.289809] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15346.688402] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15356.463621] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15357.122721] Lustre: lustre-OST0000: new disk, initializing [15357.141787] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15357.157511] Lustre: 466288:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15357.434383] Lustre: 466293:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.127@tcp: can't add NID 192.168.202.127@tcp130: rc = -75 [15357.453215] Lustre: 466293:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15357.589137] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15363.232444] Lustre: 466449:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.127@tcp: can't add NID 192.168.202.127@tcp130: rc = -75 [15363.237516] Lustre: 466449:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15363.382459] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15363.404886] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15363.585989] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15368.150364] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [15379.969668] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15389.159673] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15389.162667] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15389.171221] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15389.182169] Lustre: Skipped 1 previous similar message [15394.291772] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15394.301491] Lustre: Skipped 1 previous similar message [15394.812282] Lustre: server umount lustre-OST0000 complete [15398.890762] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15398.899953] LustreError: Skipped 1 previous similar message [15398.904899] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15398.916064] Lustre: Skipped 1 previous similar message [15398.921049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15404.513875] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15404.527261] Lustre: Skipped 3 previous similar messages [15405.243663] Lustre: server umount lustre-MDT0000 complete [15409.119275] LustreError: 464311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788639776 with bad export cookie 17103408463207966386 [15409.123274] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15409.139562] LustreError: 464311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15409.451662] Lustre: server umount lustre-MDT0001 complete [15420.319452] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [15423.211257] Key type lgssc unregistered [15423.645675] LNet: 468203:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15423.656763] LNetError: 468203:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15423.667624] LNet: Removed LNI 192.168.202.127@tcp [15424.736249] LNet: Removed LNI 192.168.202.127@tcp100 [15424.739846] LNetError: 468203:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15424.751616] LNetError: 468203:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15424.762568] LNet: Removed LNI 192.168.202.127@tcp101 [15424.768821] LNet: Removed LNI 192.168.202.127@tcp102 [15424.778550] LNet: Removed LNI 192.168.202.127@tcp103 [15424.786924] LNet: Removed LNI 192.168.202.127@tcp104 [15424.789512] LNet: Removed LNI 192.168.202.127@tcp105 [15424.798624] LNet: Removed LNI 192.168.202.127@tcp106 [15424.804113] LNet: Removed LNI 192.168.202.127@tcp107 [15424.806730] LNet: Removed LNI 192.168.202.127@tcp108 [15424.812592] LNet: Removed LNI 192.168.202.127@tcp109 [15424.816557] LNet: Removed LNI 192.168.202.127@tcp110 [15424.823594] LNet: Removed LNI 192.168.202.127@tcp111 [15424.829775] LNet: Removed LNI 192.168.202.127@tcp112 [15424.844433] LNet: Removed LNI 192.168.202.127@tcp113 [15424.850383] LNet: Removed LNI 192.168.202.127@tcp114 [15424.853355] LNet: Removed LNI 192.168.202.127@tcp115 [15424.860171] LNet: Removed LNI 192.168.202.127@tcp116 [15424.862555] LNet: Removed LNI 192.168.202.127@tcp117 [15424.866392] LNet: Removed LNI 192.168.202.127@tcp118 [15424.868854] LNet: Removed LNI 192.168.202.127@tcp119 [15424.876965] LNet: Removed LNI 192.168.202.127@tcp120 [15424.879974] LNet: Removed LNI 192.168.202.127@tcp121 [15424.882830] LNet: Removed LNI 192.168.202.127@tcp122 [15424.885456] LNet: Removed LNI 192.168.202.127@tcp123 [15424.887777] LNet: Removed LNI 192.168.202.127@tcp124 [15424.889866] LNet: Removed LNI 192.168.202.127@tcp125 [15424.892307] LNet: Removed LNI 192.168.202.127@tcp126 [15424.894887] LNet: Removed LNI 192.168.202.127@tcp127 [15424.897119] LNet: Removed LNI 192.168.202.127@tcp128 [15424.899367] LNet: Removed LNI 192.168.202.127@tcp129 [15424.901643] LNet: Removed LNI 192.168.202.127@tcp130 [15424.903618] LNet: Removed LNI 192.168.202.127@tcp131 [15424.905619] LNet: Removed LNI 192.168.202.127@tcp132 [15424.906914] LNet: Removed LNI 192.168.202.127@tcp133 [15424.908416] LNet: Removed LNI 192.168.202.127@tcp134 [15424.922073] LNet: Removed LNI 192.168.202.127@tcp135 [15425.927417] Key type .llcrypt unregistered [15425.930677] Key type ._llcrypt unregistered [15451.757443] Key type ._llcrypt registered [15451.760893] Key type .llcrypt registered [15451.873032] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_hostid [15467.199241] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [15468.203529] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15468.250507] alg: No test for adler32 (adler32-zlib) [15469.271493] Lustre: Lustre: Build Version: 2.17.58_2_g69c2166 [15469.465662] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [15471.120187] Key type lgssc registered [15471.882963] Lustre: Echo OBD driver; http://www.lustre.org/ [15513.729450] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 16:24:39 (1788639879) [15535.086529] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [15538.576379] Key type lgssc unregistered [15538.902158] LNet: 472283:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15538.910911] LNetError: 472283:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15538.937301] LNet: Removed LNI 192.168.202.127@tcp [15539.980462] Key type .llcrypt unregistered [15539.982190] Key type ._llcrypt unregistered [15556.874752] Key type ._llcrypt registered [15556.876935] Key type .llcrypt registered [15556.977747] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [15558.024996] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15558.128088] alg: No test for adler32 (adler32-zlib) [15559.164788] Lustre: Lustre: Build Version: 2.17.58_2_g69c2166 [15559.402279] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [15561.104155] Key type lgssc registered [15562.537567] Lustre: Echo OBD driver; http://www.lustre.org/ [15571.595678] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15571.668302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15573.077595] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15573.121267] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15573.258367] Lustre: lustre-MDT0000: new disk, initializing [15573.341379] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15573.366293] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15582.673935] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [15596.991123] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15597.085764] Lustre: 473491:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15597.187249] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15597.190293] Lustre: Skipped 1 previous similar message [15597.360842] Lustre: lustre-MDT0001: new disk, initializing [15597.472358] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15597.497413] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15597.521272] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15602.246668] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [15611.683704] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15617.120118] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15624.512506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15624.833978] Lustre: lustre-OST0000: new disk, initializing [15624.837528] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15624.843602] Lustre: 475463:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15624.936827] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15632.407089] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [15634.000522] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15634.019195] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15634.202790] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15644.604586] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15649.604489] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing lnet_if_list [15655.302630] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15655.346450] LNet: Added LNI 192.168.202.127@tcp10 [8/256/0/180] [15663.010614] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15663.030202] LNet: Added LNI 192.168.202.127@tcp11 [8/256/0/180] [15667.537882] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15667.560960] LNet: Added LNI 192.168.202.127@tcp12 [8/256/0/180] [15672.831247] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15672.855226] LNet: Added LNI 192.168.202.127@tcp13 [8/256/0/180] [15677.875608] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15677.895480] LNet: Added LNI 192.168.202.127@tcp14 [8/256/0/180] [15683.932292] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15683.971334] LNet: Added LNI 192.168.202.127@tcp15 [8/256/0/180] [15690.039771] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15690.071152] LNet: Added LNI 192.168.202.127@tcp16 [8/256/0/180] [15695.840424] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15695.878342] LNet: Added LNI 192.168.202.127@tcp17 [8/256/0/180] [15702.522154] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15702.566345] LNet: Added LNI 192.168.202.127@tcp18 [8/256/0/180] [15708.122476] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15708.141483] LNet: Added LNI 192.168.202.127@tcp19 [8/256/0/180] [15713.494571] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15713.531580] LNet: Added LNI 192.168.202.127@tcp20 [8/256/0/180] [15718.848206] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15718.885869] LNet: Added LNI 192.168.202.127@tcp21 [8/256/0/180] [15724.744630] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15724.771453] LNet: Added LNI 192.168.202.127@tcp22 [8/256/0/180] [15730.037258] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15730.064823] LNet: Added LNI 192.168.202.127@tcp23 [8/256/0/180] [15735.771909] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15735.808281] LNet: Added LNI 192.168.202.127@tcp24 [8/256/0/180] [15741.537248] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15741.572209] LNet: Added LNI 192.168.202.127@tcp25 [8/256/0/180] [15747.131346] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15747.172912] LNet: Added LNI 192.168.202.127@tcp26 [8/256/0/180] [15753.352515] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15753.372682] LNet: Added LNI 192.168.202.127@tcp27 [8/256/0/180] [15759.705776] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15759.736389] LNet: Added LNI 192.168.202.127@tcp28 [8/256/0/180] [15767.771509] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15767.795629] LNet: Added LNI 192.168.202.127@tcp29 [8/256/0/180] [15775.556663] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15775.596238] LNet: Added LNI 192.168.202.127@tcp30 [8/256/0/180] [15783.138504] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15783.163806] LNet: Added LNI 192.168.202.127@tcp31 [8/256/0/180] [15790.063393] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15790.125614] LNet: Added LNI 192.168.202.127@tcp32 [8/256/0/180] [15798.578850] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15798.654752] LNet: Added LNI 192.168.202.127@tcp33 [8/256/0/180] [15807.134542] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15807.176534] LNet: Added LNI 192.168.202.127@tcp34 [8/256/0/180] [15815.811542] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15815.891496] LNet: Added LNI 192.168.202.127@tcp35 [8/256/0/180] [15824.846319] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [15824.876509] LNet: Added LNI 192.168.202.127@tcp36 [8/256/0/180] [15832.024762] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15832.062624] LNet: Added LNI 192.168.202.127@tcp37 [8/256/0/180] [15840.021575] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [15840.036377] LNet: Added LNI 192.168.202.127@tcp38 [8/256/0/180] [15849.984462] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15850.014199] LNet: Added LNI 192.168.202.127@tcp39 [8/256/0/180] [15860.180147] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [15860.216802] LNet: Added LNI 192.168.202.127@tcp40 [8/256/0/180] [15869.536565] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [15869.576469] LNet: Added LNI 192.168.202.127@tcp41 [8/256/0/180] [15880.111990] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15880.164274] LNet: Added LNI 192.168.202.127@tcp42 [8/256/0/180] [15890.214429] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15890.292472] LNet: Added LNI 192.168.202.127@tcp43 [8/256/0/180] [15900.710073] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15900.771120] LNet: Added LNI 192.168.202.127@tcp44 [8/256/0/180] [15910.760255] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15910.806602] LNet: Added LNI 192.168.202.127@tcp45 [8/256/0/180] [15919.751926] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15919.777052] LNet: Added LNI 192.168.202.127@tcp46 [8/256/0/180] [15928.938523] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15928.974372] LNet: Added LNI 192.168.202.127@tcp47 [8/256/0/180] [15939.508301] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15939.577585] LNet: Added LNI 192.168.202.127@tcp48 [8/256/0/180] [15947.958236] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15948.000031] LNet: Added LNI 192.168.202.127@tcp49 [8/256/0/180] [15957.622530] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15957.661570] LNet: Added LNI 192.168.202.127@tcp50 [8/256/0/180] [15980.520146] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15980.542060] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15980.544545] Lustre: Skipped 2 previous similar messages [15980.566064] Lustre: Skipped 2 previous similar messages [15985.798774] Lustre: server umount lustre-MDT0000 complete [15990.096750] LustreError: 473485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788640357 with bad export cookie 15857005067079513240 [15990.104942] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15990.106170] LustreError: 473485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15990.522416] Lustre: server umount lustre-MDT0001 complete [16006.117276] Lustre: 472762:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788640357/real 1788640357] req@000000004056ebd7 x1875524899239552/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788640373 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16006.154766] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16009.961484] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [16011.232094] Lustre: 472764:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788640362/real 1788640362] req@000000003254c88e x1875524899239680/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788640378 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16017.570723] Lustre: 472764:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788640368/real 1788640368] req@00000000a34c7975 x1875524899239936/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788640384 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16020.915719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16022.497358] Lustre: 472761:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788640373/real 1788640373] req@00000000ce3bef78 x1875524899240064/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788640389 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16037.264317] LustreError: 475455:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16037.292240] LustreError: 475455:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [16037.337796] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16040.358210] LustreError: 475456:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16043.184322] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [16045.543201] LustreError: 475455:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16050.660324] LustreError: 475456:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16052.265873] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16052.730802] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16057.848214] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16058.414624] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [16060.995819] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16068.109989] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16078.057113] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16103.911256] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16103.911570] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16103.923158] LustreError: Skipped 1 previous similar message [16103.935288] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16103.940456] Lustre: Skipped 1 previous similar message [16103.974728] Lustre: Skipped 1 previous similar message [16106.987377] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16106.995802] Lustre: Skipped 1 previous similar message [16112.105272] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16112.122904] Lustre: Skipped 1 previous similar message [16117.216348] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16117.221601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16117.243813] Lustre: Skipped 1 previous similar message [16117.721531] Lustre: server umount lustre-OST0000 complete [16124.387088] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16124.411205] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16127.460781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16127.475662] Lustre: Skipped 2 previous similar messages [16129.738718] Lustre: server umount lustre-MDT0000 complete [16132.581165] LustreError: 499720:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16132.604869] LustreError: 499720:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [16136.097183] LustreError: 473484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788640503 with bad export cookie 15857005067079515872 [16136.107861] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16136.111022] LustreError: 473484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16136.846666] Lustre: server umount lustre-MDT0001 complete [16152.728362] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [16156.909427] Key type lgssc unregistered [16157.378692] LNet: 502518:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16157.386625] LNetError: 502518:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16157.393287] LNet: Removed LNI 192.168.202.127@tcp [16157.396556] LNet: Removed LNI 192.168.202.127@tcp10 [16157.401963] LNet: Removed LNI 192.168.202.127@tcp11 [16157.404628] LNet: Removed LNI 192.168.202.127@tcp12 [16157.407667] LNet: Removed LNI 192.168.202.127@tcp13 [16157.410941] LNet: Removed LNI 192.168.202.127@tcp14 [16157.414782] LNet: Removed LNI 192.168.202.127@tcp15 [16157.419311] LNet: Removed LNI 192.168.202.127@tcp16 [16157.423788] LNet: Removed LNI 192.168.202.127@tcp17 [16157.427326] LNet: Removed LNI 192.168.202.127@tcp18 [16157.430178] LNet: Removed LNI 192.168.202.127@tcp19 [16157.432868] LNet: Removed LNI 192.168.202.127@tcp20 [16157.434454] LNet: Removed LNI 192.168.202.127@tcp21 [16157.435928] LNet: Removed LNI 192.168.202.127@tcp22 [16157.444880] LNet: Removed LNI 192.168.202.127@tcp23 [16157.448996] LNet: Removed LNI 192.168.202.127@tcp24 [16157.450424] LNet: Removed LNI 192.168.202.127@tcp25 [16157.457441] LNet: Removed LNI 192.168.202.127@tcp26 [16157.463609] LNet: Removed LNI 192.168.202.127@tcp27 [16157.466812] LNet: Removed LNI 192.168.202.127@tcp28 [16157.474234] LNet: Removed LNI 192.168.202.127@tcp29 [16157.481812] LNet: Removed LNI 192.168.202.127@tcp30 [16157.499055] LNet: Removed LNI 192.168.202.127@tcp31 [16157.504850] LNet: Removed LNI 192.168.202.127@tcp32 [16157.510215] LNet: Removed LNI 192.168.202.127@tcp33 [16157.518476] LNet: Removed LNI 192.168.202.127@tcp34 [16157.525208] LNet: Removed LNI 192.168.202.127@tcp35 [16157.531281] LNet: Removed LNI 192.168.202.127@tcp36 [16157.536578] LNet: Removed LNI 192.168.202.127@tcp37 [16157.542227] LNet: Removed LNI 192.168.202.127@tcp38 [16157.545480] LNet: Removed LNI 192.168.202.127@tcp39 [16157.547880] LNet: Removed LNI 192.168.202.127@tcp40 [16157.550844] LNet: Removed LNI 192.168.202.127@tcp41 [16157.553906] LNet: Removed LNI 192.168.202.127@tcp42 [16157.556351] LNet: Removed LNI 192.168.202.127@tcp43 [16157.562157] LNet: Removed LNI 192.168.202.127@tcp44 [16157.563309] LNet: Removed LNI 192.168.202.127@tcp45 [16157.565051] LNet: Removed LNI 192.168.202.127@tcp46 [16157.570505] LNet: Removed LNI 192.168.202.127@tcp47 [16157.574977] LNet: Removed LNI 192.168.202.127@tcp48 [16157.578621] LNet: Removed LNI 192.168.202.127@tcp49 [16157.587464] LNet: Removed LNI 192.168.202.127@tcp50 [16158.777654] Key type .llcrypt unregistered [16158.779364] Key type ._llcrypt unregistered [16187.872176] Key type ._llcrypt registered [16187.874206] Key type .llcrypt registered [16187.991614] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_hostid [16206.400479] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [16208.143875] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16208.270614] alg: No test for adler32 (adler32-zlib) [16209.635325] Lustre: Lustre: Build Version: 2.17.58_2_g69c2166 [16210.035495] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [16211.896226] Key type lgssc registered [16213.857821] Lustre: Echo OBD driver; http://www.lustre.org/ [16254.447279] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16254.491584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16255.825118] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16255.858379] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16255.937598] Lustre: lustre-MDT0000: new disk, initializing [16256.036485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16256.078240] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16261.343145] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [16273.265950] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16273.476154] Lustre: 505712:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16273.577853] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16273.581928] Lustre: Skipped 1 previous similar message [16273.639647] Lustre: lustre-MDT0001: new disk, initializing [16273.710241] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16273.745956] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16273.761346] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16278.370136] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [16289.841668] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16298.152364] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16309.205919] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16309.440544] Lustre: *** cfs_fail_loc=90f, val=0*** [16309.550759] Lustre: lustre-OST0000: new disk, initializing [16309.554360] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16309.562591] Lustre: 507728:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16309.701811] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16314.993113] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16315.007757] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16315.131411] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16317.826164] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [16328.308343] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16340.964427] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16340.971069] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16340.987256] LustreError: Skipped 1 previous similar message [16340.987964] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16341.017619] Lustre: Skipped 1 previous similar message [16342.632732] Lustre: server umount lustre-OST0000 complete [16350.690897] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16350.704863] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16350.724262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16350.730258] Lustre: Skipped 1 previous similar message [16353.229625] Lustre: server umount lustre-MDT0000 complete [16357.760036] LustreError: 505704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788640724 with bad export cookie 16637663589702658566 [16357.774385] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16358.065664] Lustre: server umount lustre-MDT0001 complete [16370.240813] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [16372.328725] Key type lgssc unregistered [16372.553951] LNet: 509637:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16372.561281] LNetError: 509637:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16373.630976] LNet: Removed LNI 192.168.202.127@tcp [16374.810828] Key type .llcrypt unregistered [16374.814828] Key type ._llcrypt unregistered [16400.386467] Key type ._llcrypt registered [16400.388022] Key type .llcrypt registered [16400.532196] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [16402.351166] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16402.395278] alg: No test for adler32 (adler32-zlib) [16403.613799] Lustre: Lustre: Build Version: 2.17.58_2_g69c2166 [16403.921036] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [16405.600158] Key type lgssc registered [16407.081674] Lustre: Echo OBD driver; http://www.lustre.org/ [16415.934956] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16416.007783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16417.411563] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16421.435980] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [16427.051475] LustreError: 510903:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16427.065116] LustreError: 510903:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [16432.107470] LustreError: 510902:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16432.952090] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16433.502093] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16441.973961] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [16452.250534] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16457.709481] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16465.405051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16465.792343] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16472.895723] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [16480.786563] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16496.082485] loop0: detected capacity change from 0 to 204800000 [16496.360209] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16496.375325] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16496.388595] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16502.645977] loop0: detected capacity change from 0 to 204800000 [16502.822154] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16502.830487] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16502.841134] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16508.181440] loop0: detected capacity change from 0 to 204800000 [16515.680135] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16515.928498] Lustre: fs15246-OST0000: new disk, initializing [16515.939792] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [16515.967120] Lustre: 514497:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16516.030436] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16523.262186] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [16535.220852] loop1: detected capacity change from 0 to 204800000 [16541.496766] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16541.633325] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [16541.718617] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [16541.773586] Lustre: fs15246-MDT0000: new disk, initializing [16541.907168] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16541.940420] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16542.090967] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [16543.137747] Lustre: 515622:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000001200b431 x1875525784095232/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788640926 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [16546.442869] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [16552.290654] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [16552.298081] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [16552.308392] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [16572.408719] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16572.438446] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16576.772748] Lustre: server umount fs15246-MDT0000 complete [16577.521069] LustreError: 510897:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16582.630067] LustreError: 510899:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16587.249907] Lustre: server umount fs15246-OST0000 complete [16601.070618] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16601.080720] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16601.089854] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16603.106400] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16603.107066] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16606.788311] Lustre: server umount lustre-OST0000 complete [16612.835930] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16612.858914] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16612.902582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16612.909746] Lustre: Skipped 1 previous similar message [16617.242524] Lustre: server umount lustre-MDT0000 complete [16618.481867] LustreError: 514845:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16618.519822] LustreError: 514845:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [16622.049625] LustreError: 510885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788640989 with bad export cookie 5330583176194949945 [16622.059305] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16622.065649] LustreError: 510885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16622.460455] Lustre: server umount lustre-MDT0001 complete [16635.913749] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [16639.147303] Key type lgssc unregistered [16639.610597] LNet: 518719:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16639.632237] LNetError: 518719:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16640.703567] LNet: Removed LNI 192.168.202.127@tcp [16641.671774] Key type .llcrypt unregistered [16641.675552] Key type ._llcrypt unregistered [16662.622530] Key type ._llcrypt registered [16662.624734] Key type .llcrypt registered [16662.736590] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [16663.945282] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16664.034369] alg: No test for adler32 (adler32-zlib) [16665.100948] Lustre: Lustre: Build Version: 2.17.58_2_g69c2166 [16665.336102] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [16666.968320] Key type lgssc registered [16667.861992] Lustre: Echo OBD driver; http://www.lustre.org/ [16713.554477] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_hostid [16724.333605] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [16766.963088] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16767.013481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16768.251778] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16768.301074] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16768.424117] Lustre: lustre-MDT0000: new disk, initializing [16768.519696] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16768.534217] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16772.523839] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [16782.636901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16782.750520] Lustre: 523186:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16782.791807] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16782.795286] Lustre: Skipped 1 previous similar message [16782.852591] Lustre: lustre-MDT0001: new disk, initializing [16782.932620] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16782.971089] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16782.986376] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16787.691572] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [16796.754361] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16801.909455] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16809.180314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16809.447284] Lustre: lustre-OST0000: new disk, initializing [16809.456921] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16809.469815] Lustre: 525155:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16809.616497] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16815.167298] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16815.180842] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16815.360569] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16816.966344] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [16828.680215] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16834.961815] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16835.215753] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16840.194250] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16840.555408] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16846.308716] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16846.312022] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16846.327140] LustreError: Skipped 1 previous similar message [16846.328212] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16846.378062] Lustre: Skipped 1 previous similar message [16849.662402] Lustre: server umount lustre-OST0000 complete [16856.547652] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16856.555034] Lustre: Skipped 1 previous similar message [16856.557690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16856.568800] Lustre: Skipped 2 previous similar messages [16861.679963] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16861.839602] Lustre: server umount lustre-MDT0000 complete [16866.322388] LustreError: 523180:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788641233 with bad export cookie 2010639280165279880 [16866.329198] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16866.333756] LustreError: 523180:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16866.808702] Lustre: server umount lustre-MDT0001 complete [16897.018087] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_hostid [16907.685896] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [16943.165110] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16633 sec ========= 16:48:28 (1788641308) [16945.244151] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 16:48:30 (1788641310) === [16949.785405] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 16:48:34 (1788641314) === [17001.385677] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [17004.083670] Key type lgssc unregistered [17004.307608] LNet: 531806:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [17004.317478] LNetError: 531806:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [17004.336575] LNet: Removed LNI 192.168.202.127@tcp [17005.426758] Key type .llcrypt unregistered [17005.429568] Key type ._llcrypt unregistered