[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 839220753 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002332] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004016] kvm-guest: setup PV IPIs [ 0.006972] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008013] pid_max: default: 32768 minimum: 301 [ 0.009126] LSM: Security Framework initializing [ 0.010063] Yama: becoming mindful. [ 0.011043] SELinux: Initializing. [ 0.012080] *** VALIDATE selinux *** [ 0.020577] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025551] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026146] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027133] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028116] *** VALIDATE tmpfs *** [ 0.030467] *** VALIDATE proc *** [ 0.032125] *** VALIDATE cgroup *** [ 0.033010] *** VALIDATE cgroup2 *** [ 0.034265] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035155] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037031] Spectre V2 : User space: Vulnerable [ 0.038011] Speculative Store Bypass: Vulnerable [ 0.041207] debug: unmapping init [mem 0xffffffffbc459000-0xffffffffbc460fff] [ 0.043260] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044758] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045022] ... version: 2 [ 0.046013] ... bit width: 48 [ 0.047014] ... generic registers: 4 [ 0.048011] ... value mask: 0000ffffffffffff [ 0.049013] ... max period: 00007fffffffffff [ 0.050015] ... fixed-purpose events: 3 [ 0.051013] ... event mask: 000000070000000f [ 0.052255] rcu: Hierarchical SRCU implementation. [ 0.054495] smp: Bringing up secondary CPUs ... [ 0.055626] x86: Booting SMP configuration: [ 0.056085] .... node #0, CPUs: #1 #2 #3 [ 0.063075] smp: Brought up 1 node, 4 CPUs [ 0.065011] smpboot: Max logical packages: 1 [ 0.066010] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.125011] node 0 deferred pages initialised in 57ms [ 0.129031] devtmpfs: initialized [ 0.130499] x86/mm: Memory block size: 128MB [ 0.133015] gcov: version magic: 0x41383552 [ 0.135347] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.141098] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.144291] pinctrl core: initialized pinctrl subsystem [ 0.146180] [ 0.146930] ************************************************************* [ 0.150013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.153013] ** ** [ 0.157013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.160021] ** ** [ 0.163013] ** This means that this kernel is built to expose internal ** [ 0.166021] ** IOMMU data structures, which may compromise security on ** [ 0.169014] ** your system. ** [ 0.172014] ** ** [ 0.176013] ** If you see this message and you are not debugging the ** [ 0.179015] ** kernel, report this immediately to your vendor! ** [ 0.183013] ** ** [ 0.186095] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.189016] ************************************************************* [ 0.194296] NET: Registered protocol family 16 [ 0.198000] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.201062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.206065] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.211086] cpuidle: using governor menu [ 0.216108] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.219512] PCI: Using configuration type 1 for base access [ 0.221123] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.227482] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.229020] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.233383] cryptd: max_cpu_qlen set to 1000 [ 0.234211] ACPI: Added _OSI(Module Device) [ 0.235093] ACPI: Added _OSI(Processor Device) [ 0.238017] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.240013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.245481] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.252684] ACPI: Interpreter enabled [ 0.254088] ACPI: PM: (supports S0 S3 S4 S5) [ 0.256014] ACPI: Using IOAPIC for interrupt routing [ 0.259097] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.266439] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.408734] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.411157] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.414021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.418093] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.424611] acpiphp: Slot [2] registered [ 0.427163] acpiphp: Slot [5] registered [ 0.428141] acpiphp: Slot [6] registered [ 0.430109] acpiphp: Slot [7] registered [ 0.432118] acpiphp: Slot [8] registered [ 0.433140] acpiphp: Slot [9] registered [ 0.436295] acpiphp: Slot [10] registered [ 0.438131] acpiphp: Slot [3] registered [ 0.440119] acpiphp: Slot [4] registered [ 0.442100] acpiphp: Slot [11] registered [ 0.446121] acpiphp: Slot [12] registered [ 0.448132] acpiphp: Slot [13] registered [ 0.451137] acpiphp: Slot [14] registered [ 0.453119] acpiphp: Slot [15] registered [ 0.455119] acpiphp: Slot [16] registered [ 0.457093] acpiphp: Slot [17] registered [ 0.458240] acpiphp: Slot [18] registered [ 0.460245] acpiphp: Slot [19] registered [ 0.462114] acpiphp: Slot [20] registered [ 0.464108] acpiphp: Slot [21] registered [ 0.466101] acpiphp: Slot [22] registered [ 0.468101] acpiphp: Slot [23] registered [ 0.470105] acpiphp: Slot [24] registered [ 0.471118] acpiphp: Slot [25] registered [ 0.473102] acpiphp: Slot [26] registered [ 0.475117] acpiphp: Slot [27] registered [ 0.477098] acpiphp: Slot [28] registered [ 0.478100] acpiphp: Slot [29] registered [ 0.480119] acpiphp: Slot [30] registered [ 0.483104] acpiphp: Slot [31] registered [ 0.485161] PCI host bridge to bus 0000:00 [ 0.487015] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.490021] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.494020] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.497021] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.500020] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.503025] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.505165] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.509114] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.512351] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.523013] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.529056] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.531016] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.534015] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.537017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.542011] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.544833] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.548110] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.550824] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.556012] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.571015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.577013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.583164] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.592019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.603014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.631015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.645047] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.653018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.663014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.704025] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.714949] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.719035] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.725018] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.743063] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.756668] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.763030] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.772020] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.801026] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.827531] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.840028] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.855015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.881020] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.895843] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.901023] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.904928] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.921036] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.932158] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.935490] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.939470] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.942660] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.946429] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.955119] iommu: Default domain type: Passthrough [ 0.959425] SCSI subsystem initialized [ 0.962190] ACPI: bus type USB registered [ 0.965179] usbcore: registered new interface driver usbfs [ 0.968145] usbcore: registered new interface driver hub [ 0.973315] usbcore: registered new device driver usb [ 0.976214] pps_core: LinuxPPS API ver. 1 registered [ 0.979020] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.983118] PTP clock support registered [ 0.986123] EDAC MC: Ver: 3.0.0 [ 0.989179] PCI: Using ACPI for IRQ routing [ 0.992188] NetLabel: Initializing [ 0.994015] NetLabel: domain hash size = 128 [ 0.996017] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.998210] NetLabel: unlabeled traffic allowed by default [ 1.001454] vgaarb: loaded [ 1.003673] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.006053] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.013335] clocksource: Switched to clocksource kvm-clock [ 1.143874] VFS: Disk quotas dquot_6.6.0 [ 1.145647] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.148619] *** VALIDATE ramfs *** [ 1.150150] *** VALIDATE hugetlbfs *** [ 1.152102] pnp: PnP ACPI init [ 1.154918] pnp: PnP ACPI: found 6 devices [ 1.172594] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.177204] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.180182] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.183128] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.186447] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.189239] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.192941] NET: Registered protocol family 2 [ 1.195836] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.201583] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.205883] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.212365] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.216760] TCP: Hash tables configured (established 65536 bind 65536) [ 1.219850] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.222638] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.225089] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.227514] NET: Registered protocol family 1 [ 1.229988] RPC: Registered named UNIX socket transport module. [ 1.232867] RPC: Registered udp transport module. [ 1.234319] RPC: Registered tcp transport module. [ 1.235704] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.238270] NET: Registered protocol family 44 [ 1.239568] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.241179] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.242902] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.245265] PCI: CLS 0 bytes, default 64 [ 1.248341] Unpacking initramfs... [ 2.852424] debug: unmapping init [mem 0xffff8d473cc54000-0xffff8d473ffbffff] [ 2.858632] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.860673] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.864631] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.481815] Initialise system trusted keyrings [ 3.484209] Key type blacklist registered [ 3.486830] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.498123] zbud: loaded [ 3.501387] *** VALIDATE nfs *** [ 3.503011] *** VALIDATE nfs4 *** [ 3.504989] pstore: using deflate compression [ 3.509375] Platform Keyring initialized [ 3.628776] NET: Registered protocol family 38 [ 3.634410] Key type asymmetric registered [ 3.636802] Asymmetric key parser 'x509' registered [ 3.640379] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.645678] io scheduler mq-deadline registered [ 3.647870] io scheduler kyber registered [ 3.649324] io scheduler bfq registered [ 3.652167] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.656191] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.659715] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.666393] ACPI: Power Button [PWRF] [ 3.677682] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.689570] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.719703] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.738460] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.784601] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.818176] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.849829] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.856151] Non-volatile memory driver v1.3 [ 3.858688] Linux agpgart interface v0.103 [ 3.890685] virtio_blk virtio1: [vda] 146672 512-byte logical blocks (75.1 MB/71.6 MiB) [ 3.896471] vda: detected capacity change from 0 to 75096064 [ 3.914992] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.918367] vdb: detected capacity change from 0 to 1073741824 [ 3.942216] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.946107] vdc: detected capacity change from 0 to 2621440000 [ 3.961398] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.965062] vdd: detected capacity change from 0 to 2621440000 [ 3.984227] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.987477] vde: detected capacity change from 0 to 4294967296 [ 4.003553] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.006987] vdf: detected capacity change from 0 to 4294967296 [ 4.014665] libphy: Fixed MDIO Bus: probed [ 4.024186] usbcore: registered new interface driver usbserial_generic [ 4.027586] usbserial: USB Serial support registered for generic [ 4.031212] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.037256] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.038932] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.042152] mousedev: PS/2 mouse device common for all mice [ 4.045951] rtc_cmos 00:05: RTC can wake from S4 [ 4.049187] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.054541] rtc_cmos 00:05: registered as rtc0 [ 4.056729] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.065179] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.065846] intel_pstate: CPU model not supported [ 4.078181] hid: raw HID events driver (C) Jiri Kosina [ 4.082859] usbcore: registered new interface driver usbhid [ 4.086821] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.089254] usbhid: USB HID core driver [ 4.089461] drop_monitor: Initializing network drop monitor service [ 4.089593] Initializing XFRM netlink socket [ 4.091189] NET: Registered protocol family 10 [ 4.113450] Segment Routing with IPv6 [ 4.115529] NET: Registered protocol family 17 [ 4.118320] mpls_gso: MPLS GSO support [ 4.127066] RAS: Correctable Errors collector initialized. [ 4.130127] AVX version of gcm_enc/dec engaged. [ 4.132417] AES CTR mode by8 optimization enabled [ 4.232503] sched_clock: Marking stable (4232402533, 0)->(5425131704, -1192729171) [ 4.240720] registered taskstats version 1 [ 4.244961] Loading compiled-in X.509 certificates [ 4.249808] zswap: loaded using pool lzo/zbud [ 4.296560] Key type big_key registered [ 4.326986] Key type encrypted registered [ 4.329478] ima: No TPM chip found, activating TPM-bypass! [ 4.332330] ima: Allocated hash algorithm: sha1 [ 4.335350] ima: No architecture policies found [ 4.338521] evm: Initialising EVM extended attributes: [ 4.341533] evm: security.selinux [ 4.343412] evm: security.ima [ 4.345243] evm: security.capability [ 4.347313] evm: HMAC attrs: 0x1 [ 4.350732] rtc_cmos 00:05: setting system clock to 2026-09-13 02:02:36 UTC (1789264956) [ 4.360523] debug: unmapping init [mem 0xffffffffbd403000-0xffffffffbd5fffff] [ 4.365645] debug: unmapping init [mem 0xffffffffbc182000-0xffffffffbc458fff] [ 4.376110] Write protecting the kernel read-only data: 28672k [ 4.381281] debug: unmapping init [mem 0xffffffffba803000-0xffffffffba9fffff] [ 4.384500] debug: unmapping init [mem 0xffffffffbb114000-0xffffffffbb1fffff] [ 4.427039] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.443374] systemd[1]: Detected virtualization kvm. [ 4.445391] systemd[1]: Detected architecture x86-64. [ 4.447357] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.533119] systemd[1]: No hostname configured. [ 4.535076] systemd[1]: Set hostname to . [ 4.540248] random: systemd: uninitialized urandom read (16 bytes read) [ 4.543560] systemd[1]: Initializing machine ID from random generator. [ 4.676913] random: ln: uninitialized urandom read (6 bytes read) [ 4.807029] random: systemd: uninitialized urandom read (16 bytes read) [ 4.810972] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 4.816309] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 4.823564] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. Starting Journal Service... Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Swap. [ OK ] Reached target Timers. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Initrd Root Device. Starting Create Volatile Files and Directories... [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.585874] device-mapper: uevent: version 1.0.3 [ 7.598825] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook...[ 9.359475] virtio_net virtio0 ens2: renamed from eth0 [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK [ 9.394836] random: fast init done ] Started Hardware RNG Entropy Gatherer Daemon. [ 9.820651] scsi host0: ata_piix [ 9.825511] scsi host1: ata_piix [ 9.838145] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 9.861119] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 16.936235] random: crng init done [ 16.939167] random: 7 urandom warning(s) missed due to ratelimiting [ 20.236926] dracut-initqueue[588]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 21.610466] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 25.232834] printk: systemd: 20 output lines suppressed due to ratelimiting [ 26.207395] SELinux: Disabled at runtime. [ 26.317231] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 26.334502] systemd[1]: Detected virtualization kvm. [ 26.341954] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 28.124742] systemd[1]: initrd-switch-root.service: Succeeded. [ 28.135434] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 28.147487] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 28.154682] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 28.159061] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 28.180220] systemd[1]: Starting Journal Service... Starting Journal Service... [ 28.209340] systemd[1]: Starting Remount Root and Kernel File Systems... Starting Remount Root and Kernel File Systems... [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Create list of required st…ce nodes for the current kernel... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK [ 28.382790] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS ] Listening on udev Control Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-getty.slice. [ OK ] Reached target Paths. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target Local Encrypted Volumes. Mounting Kernel Debug File System... [ OK ] Listening on Process Core Dump Socket. Starting Apply Kernel Variables... Mounting Huge Pages File System... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Mounting POSIX Message Queue File System... [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-serial\x2dgetty.slice. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 29.786201] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 31.279914] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 31.478464] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 33.025761] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 33.126151] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit)[ 37.537302] Key type dns_resolver registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit)[ 38.602699] NFS: Registering the id_resolver key type [ 38.607693] Key type id_resolver registered [ 38.611894] Key type id_legacy registered [ ***] A start job is running for Configur…only root support (11s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... Starting Network Manager... [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Login Service. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg305-server login: [ 107.958674] libcfs: loading out-of-tree module taints kernel. [ 108.001906] Key type ._llcrypt registered [ 108.004607] Key type .llcrypt registered [ 108.120992] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_hostid [ 126.405868] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 126.720008] hrtimer: interrupt took 2018701 ns [ 127.974394] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 127.983818] alg: No test for adler32 (adler32-zlib) [ 129.476598] Lustre: Lustre: Build Version: 2.17.57_105_g2595dd0 [ 130.426749] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 132.143254] Key type lgssc registered [ 133.546594] Lustre: Echo OBD driver; http://www.lustre.org/ [ 150.007868] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 188.092882] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 200.031768] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 200.075921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 201.346084] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 201.390719] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 201.494612] Lustre: lustre-MDT0000: new disk, initializing [ 201.566959] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 201.595632] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 205.842625] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 218.115877] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 218.241442] Lustre: 6508:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 218.269694] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 218.278237] Lustre: Skipped 1 previous similar message [ 218.386361] Lustre: lustre-MDT0001: new disk, initializing [ 218.443490] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 218.485127] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 218.499778] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 222.773534] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 226.973552] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 235.330043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 235.581993] Lustre: lustre-OST0000: new disk, initializing [ 235.587360] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 235.595901] Lustre: 8442:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 235.646093] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 241.559870] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 244.769407] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 244.776196] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 244.824441] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 254.689977] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 254.858612] Lustre: lustre-OST0001: new disk, initializing [ 254.867379] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 254.872563] Lustre: 9516:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 254.934104] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 262.199662] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 262.210148] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 262.326654] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 262.415294] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 275.183132] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 280.911464] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 287.953951] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing check_logdir /tmp/testlogs/ [ 292.847895] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing yml_node [ 297.007516] Lustre: DEBUG MARKER: Client: 2.17.57.105 [ 299.364088] Lustre: DEBUG MARKER: MDS: 2.17.57.105 [ 301.586800] Lustre: DEBUG MARKER: OSS: 2.17.57.105 [ 303.173480] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Sep 12 22:07:33 EDT 2026 [ 319.576957] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 321.124566] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 339.426938] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 339.431351] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 339.442918] Lustre: Skipped 3 previous similar messages [ 339.457094] Lustre: Skipped 3 previous similar messages [ 343.042732] Lustre: server umount lustre-MDT0000 complete [ 346.850732] LustreError: 6499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789265298 with bad export cookie 13187160437889349681 [ 346.869220] LustreError: 6499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 346.880899] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 349.663839] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 349.665461] LustreError: 6515:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 349.669540] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 349.688247] Lustre: Skipped 1 previous similar message [ 349.715019] LustreError: 6515:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 351.712292] LustreError: 6513:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 351.712366] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 351.748341] LustreError: 6513:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 351.769293] Lustre: Skipped 2 previous similar messages [ 353.466216] Lustre: server umount lustre-MDT0001 complete [ 357.903033] Lustre: server umount lustre-OST0000 complete [ 362.507593] Lustre: server umount lustre-OST0001 complete [ 369.868687] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_hostid [ 380.820331] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 419.943507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 420.399487] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 420.476695] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 420.645535] Lustre: lustre-MDT0000: new disk, initializing [ 420.794384] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 420.810144] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 426.386129] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 436.995847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 437.141657] Lustre: 15071:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 437.197179] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 437.201345] Lustre: Skipped 1 previous similar message [ 437.275630] Lustre: lustre-MDT0001: new disk, initializing [ 437.395288] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 437.499081] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 437.518743] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 441.871199] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 451.881338] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 457.796499] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 464.934440] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 465.235313] Lustre: lustre-OST0000: new disk, initializing [ 465.241192] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 465.247897] Lustre: 17040:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 465.347838] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 467.034966] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 467.049458] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 467.202449] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 471.549835] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 480.657907] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 485.336853] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 485.551913] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 489.910315] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 490.145517] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 493.024532] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 493.025100] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 493.038824] LustreError: Skipped 1 previous similar message [ 493.042943] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 493.065409] Lustre: Skipped 1 previous similar message [ 493.097914] Lustre: Skipped 1 previous similar message [ 498.150023] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 498.162604] Lustre: Skipped 1 previous similar message [ 506.335139] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 506.530524] Lustre: server umount lustre-OST0000 complete [ 513.514535] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 513.525480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 513.546780] Lustre: Skipped 1 previous similar message [ 513.547047] Lustre: Skipped 2 previous similar messages [ 516.848654] Lustre: server umount lustre-MDT0000 complete [ 518.628600] LustreError: 15083:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 518.650651] LustreError: 15083:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 520.866102] LustreError: 15062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789265473 with bad export cookie 13187160437889351214 [ 520.878734] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 520.886332] LustreError: 15062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 521.376120] Lustre: server umount lustre-MDT0001 complete [ 525.850521] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 22:11:16 (1789265476) [ 528.348406] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 530.268876] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 22:11:20 (1789265480) [ 536.885394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 537.300073] LustreError: 19450:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 537.364709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 541.760361] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 542.693255] LustreError: 19451:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 550.465114] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 555.659275] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 564.217893] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 570.395616] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 577.343400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 584.723191] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 592.580429] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 605.671028] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 605.692415] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 605.696523] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 605.725409] Lustre: Skipped 2 previous similar messages [ 609.198752] Lustre: server umount lustre-OST0000 complete [ 615.906597] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 615.928963] Lustre: Skipped 2 previous similar messages [ 619.205905] Lustre: server umount lustre-MDT0000 complete [ 621.030047] LustreError: 19447:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 621.067417] LustreError: 19447:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 622.725351] LustreError: 19430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789265574 with bad export cookie 13187160437889352089 [ 622.731656] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 622.738376] LustreError: 19430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 623.053432] Lustre: server umount lustre-MDT0001 complete [ 632.874728] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 635.615823] Key type lgssc unregistered [ 635.889590] LNet: 22981:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 635.901115] LNetError: 22981:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 635.922297] LNet: Removed LNI 192.168.203.105@tcp [ 636.721153] Key type .llcrypt unregistered [ 636.723971] Key type ._llcrypt unregistered [ 658.286645] Key type ._llcrypt registered [ 658.289342] Key type .llcrypt registered [ 658.415709] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 659.275097] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 659.326321] alg: No test for adler32 (adler32-zlib) [ 660.489903] Lustre: Lustre: Build Version: 2.17.57_105_g2595dd0 [ 660.758275] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 662.423166] Key type lgssc registered [ 663.507815] Lustre: Echo OBD driver; http://www.lustre.org/ [ 671.741239] loop: module loaded [ 671.798085] loop0: detected capacity change from 0 to 204800000 [ 671.979654] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 671.989756] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 672.355405] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 678.952548] loop0: detected capacity change from 0 to 204800000 [ 679.160021] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 679.165704] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 679.180505] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 685.038317] loop0: detected capacity change from 0 to 204800000 [ 689.979513] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 690.036171] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 691.379380] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 691.458406] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 691.518271] Lustre: lustre-MDT0000: new disk, initializing [ 691.572350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 691.590967] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 696.975977] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 706.945816] loop1: detected capacity change from 0 to 204800000 [ 714.310250] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 714.576818] Lustre: lustre-OST07c6: new disk, initializing [ 714.579253] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 714.584387] Lustre: Skipped 1 previous similar message [ 714.588599] Lustre: 26165:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 714.636259] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 720.171225] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 720.578411] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 720.594088] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 720.628957] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 725.842239] Lustre: 25154:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 725.852338] Lustre: 25154:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 725.857293] Lustre: 25154:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 725.866804] Lustre: 25154:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 725.874625] Lustre: 25154:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 725.883507] Lustre: 25154:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 742.495828] Lustre: 26682:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 742.504435] Lustre: 26682:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 742.508609] Lustre: 26682:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 742.512857] Lustre: 26682:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 742.520934] Lustre: 26682:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 742.526994] Lustre: 26682:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 747.357538] Lustre: 25153:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 747.374153] Lustre: 25153:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 747.394972] Lustre: 25153:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 747.407849] Lustre: 25153:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 747.414894] Lustre: 25153:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 747.427446] Lustre: 25153:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 747.436263] Lustre: 25153:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 747.452261] Lustre: 25153:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 747.459447] Lustre: 25153:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 747.476581] Lustre: 25153:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 747.484401] Lustre: 25153:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 747.489740] Lustre: 25153:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 752.554546] Lustre: 26682:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 752.565883] Lustre: 26682:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 752.571533] Lustre: 26682:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 752.576110] Lustre: 26682:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 752.580837] Lustre: 26682:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 752.589461] Lustre: 26682:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 752.602444] Lustre: 26682:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 752.611928] Lustre: 26682:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 752.619777] Lustre: 26682:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 752.624536] Lustre: 26682:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 752.629330] Lustre: 26682:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 752.635212] Lustre: 26682:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 757.837938] Lustre: 25152:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 757.848266] Lustre: 25152:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 757.853339] Lustre: 25152:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 757.857362] Lustre: 25152:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 757.862492] Lustre: 25152:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 757.868019] Lustre: 25152:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 757.872989] Lustre: 25152:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 757.877479] Lustre: 25152:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 757.881296] Lustre: 25152:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 757.884391] Lustre: 25152:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 757.888876] Lustre: 25152:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 757.894158] Lustre: 25152:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 769.033984] Lustre: 25153:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 769.041262] Lustre: 25153:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 769.045970] Lustre: 25153:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 769.054193] Lustre: 25153:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 769.061041] Lustre: 25153:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 769.069389] Lustre: 25153:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 769.077252] Lustre: 25153:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 769.082294] Lustre: 25153:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 769.089321] Lustre: 25153:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 769.100735] Lustre: 25153:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 769.109253] Lustre: 25153:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 769.114300] Lustre: 25153:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 818.146466] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 818.152743] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 818.169153] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 823.455218] Lustre: server umount lustre-OST07c6 complete [ 833.494691] Lustre: server umount lustre-MDT0000 complete [ 852.666336] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 856.063755] Key type lgssc unregistered [ 856.388446] LNet: 28731:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 856.400727] LNetError: 28731:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 856.419971] LNet: Removed LNI 192.168.203.105@tcp [ 857.353246] Key type .llcrypt unregistered [ 857.356393] Key type ._llcrypt unregistered [ 877.255874] Key type ._llcrypt registered [ 877.263882] Key type .llcrypt registered [ 877.441251] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 878.175404] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 878.260046] alg: No test for adler32 (adler32-zlib) [ 879.473162] Lustre: Lustre: Build Version: 2.17.57_105_g2595dd0 [ 879.780467] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 881.423154] Key type lgssc registered [ 882.543768] Lustre: Echo OBD driver; http://www.lustre.org/ [ 890.975825] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 891.023170] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 892.545186] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 897.936739] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 902.112715] LustreError: 29995:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 902.139356] LustreError: 29995:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 907.233623] LustreError: 29996:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 908.100166] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 908.538759] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 913.181387] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 921.438581] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 927.861852] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 935.699952] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 936.075651] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 937.078942] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 942.741711] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 951.086949] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 968.170137] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 968.179417] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 968.189774] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 969.184572] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 969.188996] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 969.199478] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 969.216533] Lustre: Skipped 1 previous similar message [ 973.941460] Lustre: server umount lustre-OST0000 complete [ 979.436112] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 979.447534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 979.462442] Lustre: Skipped 1 previous similar message [ 979.472777] Lustre: Skipped 1 previous similar message [ 983.812772] Lustre: server umount lustre-MDT0000 complete [ 984.547266] LustreError: 29996:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 984.583787] LustreError: 29996:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 987.398901] LustreError: 29978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789265939 with bad export cookie 5298174784560784182 [ 987.407913] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 987.410315] LustreError: 29978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 987.874498] Lustre: server umount lustre-MDT0001 complete [ 999.063791] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 1002.084974] Key type lgssc unregistered [ 1002.347702] LNet: 33735:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1002.355564] LNetError: 33735:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1002.369536] LNet: Removed LNI 192.168.203.105@tcp [ 1003.238507] Key type .llcrypt unregistered [ 1003.240798] Key type ._llcrypt unregistered [ 1022.358870] Key type ._llcrypt registered [ 1022.376115] Key type .llcrypt registered [ 1022.496309] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 1023.638779] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1024.040062] alg: No test for adler32 (adler32-zlib) [ 1025.158685] Lustre: Lustre: Build Version: 2.17.57_105_g2595dd0 [ 1025.382158] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 1027.106834] Key type lgssc registered [ 1029.277631] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1042.065451] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1042.113385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1043.659703] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1048.613214] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 1053.169383] LustreError: 35001:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1053.191119] LustreError: 35001:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1056.908896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1057.351341] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1062.417393] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 1071.409524] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1078.033502] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1085.719152] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1086.226740] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1088.244616] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1092.034684] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 1099.125547] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1113.058551] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1113.061880] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1113.079219] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1113.101476] Lustre: Skipped 1 previous similar message [ 1118.179869] LustreError: 36963:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1118.210612] LustreError: 36963:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1118.272057] Lustre: server umount lustre-OST0000 complete [ 1123.298735] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1123.305429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1123.321629] Lustre: Skipped 2 previous similar messages [ 1123.332153] Lustre: Skipped 1 previous similar message [ 1128.417357] LustreError: 35002:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1128.439739] LustreError: 35002:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1128.477381] Lustre: server umount lustre-MDT0000 complete [ 1132.169222] LustreError: 36845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789266084 with bad export cookie 1380728661848799346 [ 1132.189169] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1132.547616] Lustre: server umount lustre-MDT0001 complete [ 1144.427566] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 1147.972368] Key type lgssc unregistered [ 1148.322758] LNet: 38557:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1148.332788] LNetError: 38557:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1148.345647] LNet: Removed LNI 192.168.203.105@tcp [ 1149.083748] Key type .llcrypt unregistered [ 1149.086825] Key type ._llcrypt unregistered [ 1172.629623] Key type ._llcrypt registered [ 1172.632159] Key type .llcrypt registered [ 1172.711397] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 1173.792758] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1173.830175] alg: No test for adler32 (adler32-zlib) [ 1174.851407] Lustre: Lustre: Build Version: 2.17.57_105_g2595dd0 [ 1175.049882] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 1176.714081] Key type lgssc registered [ 1177.778805] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1185.118752] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1185.150223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1186.648340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1191.145704] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 1196.007634] LustreError: 39971:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1196.031279] LustreError: 39971:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1198.758163] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1199.110973] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1202.945672] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 1209.556441] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1216.804934] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1225.057480] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1225.555486] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1231.727218] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1232.200573] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 1239.671816] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1250.272539] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1250.282545] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1250.306610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1252.326393] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1252.332616] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1252.349655] Lustre: Skipped 1 previous similar message [ 1252.367580] Lustre: Skipped 2 previous similar messages [ 1256.513446] Lustre: server umount lustre-MDT0000 complete [ 1257.441992] LustreError: 41026:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1257.469446] LustreError: 41026:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1260.586545] LustreError: 39951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789266212 with bad export cookie 2723024623855419285 [ 1260.599675] LustreError: 39951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1260.600653] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1262.560322] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1262.561707] LustreError: 41811:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1262.593211] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1262.609135] LustreError: 41811:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1262.632747] Lustre: Skipped 1 previous similar message [ 1265.568821] LustreError: 39966:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1265.578604] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1265.580520] LustreError: 39966:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1265.598961] Lustre: Skipped 1 previous similar message [ 1266.975020] Lustre: server umount lustre-MDT0001 complete [ 1281.949232] Lustre: server umount lustre-OST0000 complete [ 1296.937816] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 1299.688490] Key type lgssc unregistered [ 1300.013552] LNet: 43828:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1300.023634] LNetError: 43828:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1300.040488] LNet: Removed LNI 192.168.203.105@tcp [ 1300.958626] Key type .llcrypt unregistered [ 1300.961522] Key type ._llcrypt unregistered [ 1331.335171] Key type ._llcrypt registered [ 1331.336672] Key type .llcrypt registered [ 1331.481734] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 1332.729540] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1332.763367] alg: No test for adler32 (adler32-zlib) [ 1333.818304] Lustre: Lustre: Build Version: 2.17.57_105_g2595dd0 [ 1334.013080] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 1335.663454] Key type lgssc registered [ 1337.162898] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1346.176527] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1346.195227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1347.838579] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1353.580816] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 1357.285063] LustreError: 45242:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1357.317722] LustreError: 45242:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1362.403832] LustreError: 45243:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1362.840985] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1363.293926] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1368.460570] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 1379.329169] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1386.631410] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1396.125706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1396.594871] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1403.524873] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 1412.633528] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1417.078283] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1423.815415] Lustre: lustre-OST0000: Not available for connect from 192.168.203.5@tcp (stopping) [ 1424.352713] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1424.373377] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1424.412073] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1427.428432] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1427.437150] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1427.469501] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1429.715744] Lustre: server umount lustre-OST0000 complete [ 1450.467331] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1450.486259] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1450.499611] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1450.510519] Lustre: Skipped 2 previous similar messages [ 1453.470973] Lustre: server umount lustre-MDT0000 complete [ 1457.310704] LustreError: 45222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789266409 with bad export cookie 14378539499716469788 [ 1457.320811] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1457.322280] LustreError: 45222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1463.627863] Lustre: server umount lustre-MDT0001 complete [ 1476.417907] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 1479.330183] Key type lgssc unregistered [ 1479.605699] LNet: 48955:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1479.619021] LNetError: 48955:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1479.639848] LNet: Removed LNI 192.168.203.105@tcp [ 1480.618646] Key type .llcrypt unregistered [ 1480.623453] Key type ._llcrypt unregistered [ 1506.545834] Key type ._llcrypt registered [ 1506.550314] Key type .llcrypt registered [ 1506.654851] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 1507.847102] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1507.870223] alg: No test for adler32 (adler32-zlib) [ 1508.944973] Lustre: Lustre: Build Version: 2.17.57_105_g2595dd0 [ 1509.176656] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 1510.919330] Key type lgssc registered [ 1512.645970] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1521.334851] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1521.353724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1522.944766] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1528.144806] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 1533.416480] LustreError: 50370:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1533.451034] LustreError: 50370:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1537.185225] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1537.688713] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1542.791860] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 1551.260698] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1556.205471] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1562.068326] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1562.350843] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1567.229328] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 1568.496463] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1574.076568] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1580.791619] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1583.905461] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1595.997166] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-1222h29m06s [ 1597.325515] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1599.458218] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1599.462085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1599.483397] Lustre: Skipped 1 previous similar message [ 1600.549857] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1600.558388] Lustre: Skipped 4 previous similar messages [ 1602.493708] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.5@tcp (stopping) [ 1605.480385] Lustre: server umount lustre-MDT0000 complete [ 1606.628252] LustreError: 50367:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1606.655518] LustreError: 50367:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1608.596117] LustreError: 53055:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1609.303994] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1612.802415] LustreError: 50371:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1612.831290] LustreError: 50371:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1616.769987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1616.899463] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1617.134395] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1617.191702] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1618.161441] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1621.166955] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 1624.818254] Lustre: DEBUG MARKER: Wait for df (36662) ... [ 1626.275465] Lustre: DEBUG MARKER: done [ 1632.748542] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1632.763621] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1632.783093] Lustre: Skipped 1 previous similar message [ 1632.792958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1632.804122] Lustre: Skipped 1 previous similar message [ 1633.777613] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1636.582180] Lustre: server umount lustre-OST0000 complete [ 1644.022972] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1644.033709] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1644.037630] Lustre: Skipped 1 previous similar message [ 1644.038455] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1644.056803] Lustre: Skipped 3 previous similar messages [ 1647.533535] Lustre: server umount lustre-MDT0000 complete [ 1651.175184] LustreError: 50370:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1651.196565] LustreError: 50370:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1651.676639] LustreError: 52214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789266603 with bad export cookie 16608457933787654646 [ 1651.696932] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1652.087265] Lustre: server umount lustre-MDT0001 complete [ 1664.070483] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 1666.487625] Key type lgssc unregistered [ 1666.730684] LNet: 55291:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1666.736582] LNetError: 55291:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1666.767206] LNet: Removed LNI 192.168.203.105@tcp [ 1667.699731] Key type .llcrypt unregistered [ 1667.708068] Key type ._llcrypt unregistered [ 1711.110692] Key type ._llcrypt registered [ 1711.116890] Key type .llcrypt registered [ 1711.219060] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 1712.493723] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1712.537381] alg: No test for adler32 (adler32-zlib) [ 1713.595077] Lustre: Lustre: Build Version: 2.17.57_105_g2595dd0 [ 1713.943456] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 1715.639494] Key type lgssc registered [ 1717.016572] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1725.008065] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1725.032491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1726.283850] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1726.318935] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1726.467123] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1731.361808] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 1741.688328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1741.846203] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1741.887937] Lustre: 57497:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1742.390550] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1748.988709] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 1757.501725] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1763.483633] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1770.676320] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1770.915848] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1771.237407] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1777.297244] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1777.764564] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 1785.187847] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1791.498470] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-1222h32m21s [ 1793.252363] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1800.637482] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1804.038907] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1804.091718] LustreError: 57503:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1824.191149] Lustre: lustre-MDT0000: Client 63d1bed1-00ac-49c1-970a-5b09db225e3c (at 192.168.203.5@tcp) reconnecting [ 1844.161839] LustreError: 57503:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1846.851893] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1849.748492] Lustre: DEBUG MARKER: done [ 1859.041533] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1859.055516] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1859.081167] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1859.553131] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1859.569170] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1859.585161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1862.608653] Lustre: server umount lustre-OST0000 complete [ 1869.793443] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1869.811520] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1869.814901] Lustre: Skipped 1 previous similar message [ 1872.908021] Lustre: server umount lustre-MDT0000 complete [ 1877.259948] LustreError: 57491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789266829 with bad export cookie 12394019347530502405 [ 1877.262800] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1877.277847] LustreError: 57491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1877.639167] Lustre: server umount lustre-MDT0001 complete [ 1890.181508] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 1893.240432] Key type lgssc unregistered [ 1893.519734] LNet: 61826:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1893.535254] LNetError: 61826:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1893.559153] LNet: Removed LNI 192.168.203.105@tcp [ 1894.491105] Key type .llcrypt unregistered [ 1894.493198] Key type ._llcrypt unregistered [ 1940.090456] Key type ._llcrypt registered [ 1940.094510] Key type .llcrypt registered [ 1940.223701] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 1941.512293] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1941.528703] alg: No test for adler32 (adler32-zlib) [ 1942.467121] Lustre: Lustre: Build Version: 2.17.57_105_g2595dd0 [ 1942.614774] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 1944.276555] Key type lgssc registered [ 1945.446913] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1948.981712] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1948.993885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1950.235536] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1950.249362] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1950.348093] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1951.543139] Lustre: Failing over lustre-MDT0000 [ 1952.069892] Lustre: server umount lustre-MDT0000 complete [ 1959.665280] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 22:35:09 (1789266909) [ 1966.435234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1967.158983] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1971.727690] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 1984.161518] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1984.307884] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1984.404170] Lustre: 64619:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1984.755880] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1991.056878] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2002.091204] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2008.639841] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2017.894119] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2018.209899] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2018.473879] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2025.801419] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2026.661532] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2034.815713] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2040.782629] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2051.565141] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2051.574526] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2051.591548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2052.577548] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2052.580685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2052.591911] Lustre: Skipped 1 previous similar message [ 2052.615149] Lustre: Skipped 2 previous similar messages [ 2055.189066] Lustre: server umount lustre-MDT0000 complete [ 2059.355127] LustreError: 64613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789267011 with bad export cookie 8844004012212536138 [ 2059.365677] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2059.371062] LustreError: 64613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2059.708710] Lustre: server umount lustre-MDT0001 complete [ 2063.523589] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2068.753693] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2077.773551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2078.175813] Lustre: 63540:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789267014/real 1789267014] req@000000001b8dc41f x1876182322685952/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789267030 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2078.209857] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2080.991194] Lustre: 63541:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789267017/real 1789267017] req@000000009fe90053 x1876182322686080/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789267033 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2084.319885] Lustre: 63543:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789267020/real 1789267020] req@00000000c915e50e x1876182322686208/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789267036 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2084.724089] LustreError: 66471:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2084.741893] LustreError: 66471:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2084.813123] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2084.933058] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2087.846106] LustreError: 66472:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2088.884220] LustreError: 68393:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2088.907473] LustreError: 68393:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2089.719517] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2089.889968] Lustre: 63540:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789267026/real 1789267026] req@00000000d881bd3e x1876182322686592/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789267042 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2089.918465] Lustre: 63540:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2094.060173] LustreError: 68393:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2094.075251] LustreError: 68393:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2098.535970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2099.009689] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2103.854945] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2104.294301] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2107.380460] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2113.113700] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2120.002391] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2127.840592] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2127.844697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2127.854553] Lustre: Skipped 1 previous similar message [ 2127.867336] Lustre: Skipped 2 previous similar messages [ 2132.962400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2132.972480] Lustre: Skipped 2 previous similar messages [ 2133.741749] Lustre: server umount lustre-MDT0000 complete [ 2137.611553] LustreError: 67537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789267089 with bad export cookie 8844004012212538413 [ 2137.624049] LustreError: 67537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2137.628156] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2137.913178] Lustre: server umount lustre-MDT0001 complete [ 2144.335393] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2154.163809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2154.464357] Lustre: 63540:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789267090/real 1789267090] req@00000000ff86b760 x1876182322714496/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789267106 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2154.509416] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2154.524062] Lustre: Skipped 1 previous similar message [ 2162.721722] LustreError: 63539:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000f7c5a5ab x1876182322715520/t0(0) o250->MGC192.168.203.105@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2163.138458] LustreError: 69128:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2163.231657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2163.285065] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2163.663383] Lustre: 63541:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789267099/real 1789267099] req@00000000f1f12850 x1876182322715008/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789267115 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2163.715133] Lustre: 63541:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 2167.885027] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2174.435104] LustreError: 71076:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2174.464988] LustreError: 71076:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 2177.116457] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2182.913334] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2183.933817] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2192.292214] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2198.623584] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2208.224471] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2208.232874] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2208.241906] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2211.406735] Lustre: server umount lustre-MDT0000 complete [ 2213.858208] LustreError: 66472:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2213.887726] LustreError: 66472:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2215.634964] LustreError: 64612:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789267167 with bad export cookie 8844004012212539792 [ 2215.641465] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2215.653076] LustreError: 64612:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2216.019727] Lustre: server umount lustre-MDT0001 complete [ 2223.078664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2226.918324] Lustre: DEBUG MARKER: files compared the same [ 2239.854464] Lustre: server umount lustre-OST0000 complete [ 2256.122129] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 2258.922931] Key type lgssc unregistered [ 2259.217175] LNet: 74355:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2259.233520] LNetError: 74355:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2259.260625] LNet: Removed LNI 192.168.203.105@tcp [ 2260.199523] Key type .llcrypt unregistered [ 2260.201952] Key type ._llcrypt unregistered [ 2283.298614] Key type ._llcrypt registered [ 2283.300713] Key type .llcrypt registered [ 2283.435837] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 2284.715260] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2284.828255] alg: No test for adler32 (adler32-zlib) [ 2285.971434] Lustre: Lustre: Build Version: 2.17.57_105_g2595dd0 [ 2286.310820] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 2288.007927] Key type lgssc registered [ 2289.211223] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2296.889446] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2296.904495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2298.381064] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2302.144577] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug +malloc all [ 2308.586758] LustreError: 75776:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2308.593472] LustreError: 75776:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2310.132448] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2310.451183] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2314.151867] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug +malloc all [ 2321.028815] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2326.721906] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2331.971072] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2332.174566] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2338.065070] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug +malloc all [ 2343.473140] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2348.760154] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2356.703622] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2356.713482] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2356.731468] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2358.760904] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2358.762366] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2358.789096] Lustre: Skipped 1 previous similar message [ 2362.591205] Lustre: server umount lustre-OST0000 complete [ 2366.944696] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2366.958230] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2366.986354] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2368.993439] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2368.994137] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2369.020046] Lustre: Skipped 1 previous similar message [ 2374.115049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2374.119400] Lustre: Skipped 1 previous similar message [ 2380.255878] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2380.536134] Lustre: server umount lustre-MDT0000 complete [ 2384.277422] LustreError: 75757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789267336 with bad export cookie 15484243202034205942 [ 2384.278464] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2384.292204] LustreError: 75757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2384.598259] Lustre: server umount lustre-MDT0001 complete [ 2396.588151] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 2399.279784] Key type lgssc unregistered [ 2399.587137] LNet: 79306:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2399.596056] LNetError: 79306:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2399.614642] LNet: Removed LNI 192.168.203.105@tcp [ 2400.351990] Key type .llcrypt unregistered [ 2400.355247] Key type ._llcrypt unregistered [ 2424.513426] Key type ._llcrypt registered [ 2424.515525] Key type .llcrypt registered [ 2424.593675] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 2425.411178] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2425.423119] alg: No test for adler32 (adler32-zlib) [ 2426.405530] Lustre: Lustre: Build Version: 2.17.57_105_g2595dd0 [ 2426.603162] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 2428.264358] Key type lgssc registered [ 2429.686269] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2436.912644] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2436.931153] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2452.642570] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2457.759345] LustreError: 80697:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.105@tcp: failed processing log, type 4: rc = -110 [ 2484.448286] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2489.908469] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2497.400416] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2504.645897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2504.765978] Lustre: *** cfs_fail_loc=706, val=0*** [ 2505.031381] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2505.118741] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2509.649098] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2510.307486] LustreError: 80721:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2510.325925] LustreError: 80721:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2515.437366] LustreError: 80722:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2518.404811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2518.922317] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2524.139695] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2533.376904] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2540.663565] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2544.610426] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2544.618729] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2544.637424] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2545.634929] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2545.636259] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2548.743577] Lustre: server umount lustre-OST0000 complete [ 2554.850394] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2554.861846] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2554.874992] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2554.887358] Lustre: Skipped 1 previous similar message [ 2560.992321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2561.000056] Lustre: Skipped 3 previous similar messages [ 2566.117860] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2566.123302] Lustre: Skipped 1 previous similar message [ 2567.135190] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2567.278799] Lustre: server umount lustre-MDT0000 complete [ 2570.209472] LustreError: 81786:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2570.224461] LustreError: 81786:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2571.090749] LustreError: 80704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789267523 with bad export cookie 15043907638406204756 [ 2571.113777] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2571.114943] LustreError: 80704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2571.571879] Lustre: server umount lustre-MDT0001 complete [ 2582.190907] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 2585.085028] Key type lgssc unregistered [ 2585.453624] LNet: 84312:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2585.463767] LNetError: 84312:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2585.487029] LNet: Removed LNI 192.168.203.105@tcp [ 2586.459552] Key type .llcrypt unregistered [ 2586.462270] Key type ._llcrypt unregistered [ 2606.693076] Key type ._llcrypt registered [ 2606.694710] Key type .llcrypt registered [ 2606.837956] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 2607.676904] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2607.696044] alg: No test for adler32 (adler32-zlib) [ 2608.795411] Lustre: Lustre: Build Version: 2.17.57_105_g2595dd0 [ 2609.023239] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 2610.704568] Key type lgssc registered [ 2612.002996] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2617.859068] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2617.909390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2623.716972] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2631.044907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2631.644177] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2635.891493] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2636.783324] LustreError: 86392:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2636.797605] LustreError: 86392:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2640.866812] LustreError: 86396:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2640.894712] LustreError: 86396:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2643.987416] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2644.296293] LustreError: 86395:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2644.329944] LustreError: 86395:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2644.345908] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2649.572033] LustreError: 86391:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2649.589934] LustreError: 86391:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2650.272720] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2654.690754] LustreError: 86390:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2654.704327] LustreError: 86390:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2659.097272] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2659.115021] Lustre: MGS: Client 0eed2ca5-6c01-493e-a1bb-a7f7af300a36 (at 0@lo) reconnecting [ 2659.141142] LustreError: 86406:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2659.546912] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2659.622821] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2664.461560] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2680.288301] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2680.298742] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2680.311752] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2680.320927] Lustre: Skipped 1 previous similar message [ 2684.173340] Lustre: server umount lustre-OST0000 complete [ 2690.530309] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2690.531197] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2690.535714] LustreError: Skipped 1 previous similar message [ 2690.535872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2690.535876] Lustre: Skipped 1 previous similar message [ 2690.543394] Lustre: Skipped 1 previous similar message [ 2693.452380] Lustre: server umount lustre-MDT0000 complete [ 2695.659341] LustreError: 86395:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2695.691288] LustreError: 86395:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2697.268322] Lustre: server umount lustre-MDT0001 complete [ 2700.974352] Lustre: server umount MGS complete [ 2713.229210] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 22:47:43 (1789267663) [ 2742.275572] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_hostid [ 2751.916498] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 2789.257352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2793.787585] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2803.279861] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2803.489563] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2803.537918] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2803.642174] Lustre: lustre-MDT0001: new disk, initializing [ 2803.716826] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2803.737184] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2803.747414] Lustre: 93557:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2803.757266] LustreError: 93557:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2808.140988] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2818.179488] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2818.404501] Lustre: lustre-OST0000: new disk, initializing [ 2818.414571] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2818.428057] Lustre: 94324:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2818.530657] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2824.531823] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2824.736937] Lustre: 94618:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000ce721e1f x1876183022083840/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1789267792 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2833.633504] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2840.439251] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2840.449161] Lustre: MGS: Client be9905b4-0274-4ac0-93f9-11b3e0d072df (at 0@lo) reconnecting [ 2840.501719] Lustre: 92827:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2840.527968] LustreError: 93543:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2840.545264] LustreError: 93543:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2840.560205] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2840.653248] Lustre: lustre-MDT0000: new disk, initializing [ 2840.782623] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2840.811432] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2840.836381] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2841.891113] Lustre: 95392:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000b42e8e70 x1876183022090752/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1789267810 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2845.470491] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2847.072721] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2850.999939] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2851.010792] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2851.148367] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2875.883783] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2875.904652] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2875.916691] Lustre: Skipped 1 previous similar message [ 2875.928352] Lustre: Skipped 2 previous similar messages [ 2880.996527] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2881.015586] Lustre: Skipped 1 previous similar message [ 2881.635464] Lustre: server umount lustre-OST0000 complete [ 2886.120356] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2886.129366] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2886.138132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2886.153115] Lustre: Skipped 1 previous similar message [ 2891.018080] Lustre: server umount lustre-MDT0000 complete [ 2891.235691] LustreError: 93527:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2891.258934] LustreError: 93527:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2900.834883] Lustre: server umount lustre-MDT0001 complete [ 2904.289913] Lustre: server umount MGS complete [ 2915.312121] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 22:51:05 (1789267865) [ 2930.898229] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 2934.162576] Key type lgssc unregistered [ 2934.577054] LNet: 97886:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2934.587705] LNetError: 97886:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2934.605993] LNet: Removed LNI 192.168.203.105@tcp [ 2935.636248] Key type .llcrypt unregistered [ 2935.637929] Key type ._llcrypt unregistered [ 2954.978411] Key type ._llcrypt registered [ 2954.983591] Key type .llcrypt registered [ 2955.105652] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 2956.518001] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2956.542609] alg: No test for adler32 (adler32-zlib) [ 2957.552111] Lustre: Lustre: Build Version: 2.17.57_105_g2595dd0 [ 2957.792481] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 2959.439725] Key type lgssc registered [ 2960.740123] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2969.722779] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2969.762073] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2971.503826] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2976.851961] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2981.856608] LustreError: 99105:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2981.875368] LustreError: 99105:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2985.573346] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2985.936607] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2990.634455] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2997.574971] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3003.725847] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3010.587969] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3010.905524] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3017.105647] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3018.004653] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3027.210563] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3040.760718] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3040.760942] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3040.773997] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3040.784890] Lustre: Skipped 1 previous similar message [ 3044.969744] Lustre: server umount lustre-OST0000 complete [ 3050.976402] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3050.983667] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3050.995788] Lustre: Skipped 1 previous similar message [ 3051.006404] Lustre: Skipped 1 previous similar message [ 3055.333191] Lustre: server umount lustre-MDT0000 complete [ 3056.105963] LustreError: 99101:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3056.122756] LustreError: 99101:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3059.343764] LustreError: 99841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789268011 with bad export cookie 2609292105241123574 [ 3059.345161] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3059.356660] LustreError: 99841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3059.874362] Lustre: server umount lustre-MDT0001 complete [ 3071.871982] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 3075.106456] Key type lgssc unregistered [ 3075.429122] LNet: 102639:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3075.444958] LNetError: 102639:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3075.472283] LNet: Removed LNI 192.168.203.105@tcp [ 3076.533686] Key type .llcrypt unregistered [ 3076.537467] Key type ._llcrypt unregistered [ 3094.298564] Key type ._llcrypt registered [ 3094.302896] Key type .llcrypt registered [ 3094.467713] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 3095.873074] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3095.998205] alg: No test for adler32 (adler32-zlib) [ 3097.074423] Lustre: Lustre: Build Version: 2.17.57_105_g2595dd0 [ 3097.279787] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 3098.945699] Key type lgssc registered [ 3100.070586] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3106.798936] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3106.808789] LustreError: 103721:0:(tgt_mount.c:2570:server_fill_super()) cfs_race id 716 sleeping [ 3106.865289] LustreError: 103730:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 waking [ 3106.879424] LustreError: 103721:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 awake: rc=4945 [ 3106.890922] LustreError: 103721:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3106.904319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3106.905940] LustreError: 103721:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3108.641270] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3116.875274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3117.273990] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3122.367555] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3128.919663] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing lsmod [ 3132.059712] LustreError: 104921:0:(tgt_mount.c:2570:server_fill_super()) cfs_race id 716 sleeping [ 3132.243466] LustreError: 104936:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 waking [ 3132.247728] LustreError: 104921:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 awake: rc=4831 [ 3132.252353] LustreError: 104921:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3132.257945] LustreError: 104921:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3132.274629] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3132.650267] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3133.716517] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3138.017733] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3138.031400] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3138.082345] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3139.042317] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3139.045325] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3139.056108] Lustre: Skipped 1 previous similar message [ 3139.072906] Lustre: Skipped 2 previous similar messages [ 3141.998659] Lustre: server umount lustre-MDT0000 complete [ 3142.195965] LustreError: 103767:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3142.227996] LustreError: 103767:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3146.632711] LustreError: 103748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789268098 with bad export cookie 13191241959326209732 [ 3146.652185] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3146.663310] LustreError: 103748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3147.233106] LustreError: 103762:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3147.239674] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3147.323375] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3148.511288] LustreError: 104199:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000c59e5691 x1876183533770368/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3148.737629] Lustre: server umount lustre-MDT0001 complete [ 3153.158178] Lustre: server umount lustre-OST0000 complete [ 3163.179287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3163.770363] LustreError: 106000:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3163.886221] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3169.252816] LustreError: 106001:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3169.290425] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3173.346859] LustreError: 106000:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3178.937639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3179.453468] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3184.532768] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3193.007327] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3199.681362] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3208.059404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3208.538532] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3215.143662] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3222.729870] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3233.142317] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 3239.395916] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3239.410208] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3239.429054] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3243.492333] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3243.494140] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3243.498651] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3253.220399] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3253.449906] Lustre: server umount lustre-OST0000 complete [ 3258.854560] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3258.867771] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3258.875240] Lustre: Skipped 1 previous similar message [ 3258.884201] Lustre: Skipped 3 previous similar messages [ 3271.137896] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3271.361318] Lustre: server umount lustre-MDT0000 complete [ 3274.216486] LustreError: 105995:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3274.243142] LustreError: 105995:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3275.837241] LustreError: 105982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789268227 with bad export cookie 13191241959326210663 [ 3275.846297] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3275.856221] LustreError: 105982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3276.275877] Lustre: server umount lustre-MDT0001 complete [ 3290.207162] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 3293.165153] Key type lgssc unregistered [ 3293.486814] LNet: 109558:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3293.492229] LNetError: 109558:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3294.568941] LNet: Removed LNI 192.168.203.105@tcp [ 3295.399211] Key type .llcrypt unregistered [ 3295.403426] Key type ._llcrypt unregistered [ 3320.540955] Key type ._llcrypt registered [ 3320.544187] Key type .llcrypt registered [ 3320.729086] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 3321.990072] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3322.073154] alg: No test for adler32 (adler32-zlib) [ 3323.084160] Lustre: Lustre: Build Version: 2.17.57_105_g2595dd0 [ 3323.366346] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 3325.039503] Key type lgssc registered [ 3326.035085] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3333.491625] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3333.523764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3335.031388] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3339.591457] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3345.378450] LustreError: 110971:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3345.406934] LustreError: 110971:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3347.774784] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3348.125833] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3352.537784] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3360.186542] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3366.189184] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3372.057050] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3372.492581] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3378.993923] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3380.680635] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3387.250786] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3393.077605] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3396.711896] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3401.189178] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3401.191205] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3401.194424] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3401.205376] Lustre: Skipped 1 previous similar message [ 3402.670992] Lustre: server umount lustre-OST0000 complete [ 3409.888224] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3409.896741] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3409.912388] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3409.916683] Lustre: Skipped 1 previous similar message [ 3411.424582] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3411.425853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3411.461410] Lustre: Skipped 1 previous similar message [ 3416.547335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3416.561236] Lustre: Skipped 1 previous similar message [ 3421.151121] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3421.340988] Lustre: server umount lustre-MDT0000 complete [ 3421.669116] LustreError: 113124:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3421.682858] LustreError: 113124:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3425.292192] LustreError: 113504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789268377 with bad export cookie 9998944114320597076 [ 3425.303238] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3425.755938] Lustre: server umount lustre-MDT0001 complete [ 3437.757561] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 3441.070031] Key type lgssc unregistered [ 3441.541471] LNet: 114612:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3441.557794] LNetError: 114612:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3441.583144] LNet: Removed LNI 192.168.203.105@tcp [ 3442.477662] Key type .llcrypt unregistered [ 3442.480352] Key type ._llcrypt unregistered [ 3459.983306] Key type ._llcrypt registered [ 3459.986851] Key type .llcrypt registered [ 3460.145239] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 3461.100245] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3461.224876] alg: No test for adler32 (adler32-zlib) [ 3462.345633] Lustre: Lustre: Build Version: 2.17.57_105_g2595dd0 [ 3462.554941] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 3464.183169] Key type lgssc registered [ 3465.089537] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3470.858482] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3470.871389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3472.168505] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3477.147246] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3482.601994] LustreError: 115830:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3482.623260] LustreError: 115830:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3487.240391] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3487.533125] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3492.911893] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3500.961816] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3507.245444] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3515.001601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3515.261230] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3517.298698] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3520.933907] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3528.214951] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3543.013730] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3543.025847] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3543.034121] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3543.045721] Lustre: Skipped 1 previous similar message [ 3548.130717] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3548.148143] Lustre: Skipped 2 previous similar messages [ 3548.534943] Lustre: server umount lustre-OST0000 complete [ 3554.272714] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3554.292706] LustreError: Skipped 1 previous similar message [ 3554.294742] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3554.317959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3558.373392] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3558.374105] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3560.356774] Lustre: server umount lustre-MDT0000 complete [ 3563.497077] LustreError: 116803:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3563.521850] LustreError: 116803:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3564.966804] LustreError: 115811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789268517 with bad export cookie 4401948044571625584 [ 3564.998077] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3564.999772] LustreError: 115811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3565.308568] Lustre: server umount lustre-MDT0001 complete [ 3577.347743] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 3580.558055] Key type lgssc unregistered [ 3580.875476] LNet: 119399:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3580.891328] LNetError: 119399:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3580.919521] LNet: Removed LNI 192.168.203.105@tcp [ 3581.839073] Key type .llcrypt unregistered [ 3581.841191] Key type ._llcrypt unregistered [ 3607.623371] Key type ._llcrypt registered [ 3607.626464] Key type .llcrypt registered [ 3607.751706] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 3608.585649] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3608.715666] alg: No test for adler32 (adler32-zlib) [ 3609.850948] Lustre: Lustre: Build Version: 2.17.57_105_g2595dd0 [ 3610.122754] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 3611.791163] Key type lgssc registered [ 3613.076774] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3620.989677] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3621.020245] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3622.533433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3627.046459] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3632.101922] LustreError: 120812:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3632.119596] LustreError: 120812:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3635.310573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3635.629285] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3639.494947] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3647.154985] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3653.004076] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3660.662113] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3661.253709] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3667.320306] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3667.390136] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 3676.137668] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3686.856965] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3691.758898] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3691.763205] Lustre: Skipped 2 previous similar messages [ 3694.148646] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3694.199886] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3695.984976] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3695.998426] Lustre: Skipped 2 previous similar messages [ 3699.848488] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3699.861906] Lustre: Skipped 1 previous similar message [ 3705.394957] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.5@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3705.402179] Lustre: Skipped 2 previous similar messages [ 3712.618899] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.5@tcp 192.168.0.[2,10]@tcp [ 3717.469747] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.5/24@tcp in log lustre-MDT0000 [ 3717.478098] Lustre: Skipped 2 previous similar messages [ 3725.925457] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.5/24@tcp [ 3725.930588] Lustre: Skipped 1 previous similar message [ 3732.973128] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3732.981997] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3732.997171] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3735.002158] Lustre: server umount lustre-OST0000 complete [ 3742.178619] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3742.181715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3742.203992] Lustre: Skipped 1 previous similar message [ 3742.215550] Lustre: Skipped 1 previous similar message [ 3745.254172] Lustre: server umount lustre-MDT0000 complete [ 3747.298902] LustreError: 122243:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3747.316931] LustreError: 122243:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3748.990083] LustreError: 123444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789268701 with bad export cookie 6247801807975472250 [ 3748.994196] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3749.007748] LustreError: 123444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3759.731140] Lustre: server umount lustre-MDT0001 complete [ 3770.314573] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 3772.865369] Key type lgssc unregistered [ 3773.126958] LNet: 125903:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3773.139180] LNetError: 125903:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3773.155687] LNet: Removed LNI 192.168.203.105@tcp [ 3773.895696] Key type .llcrypt unregistered [ 3773.899458] Key type ._llcrypt unregistered [ 3801.926556] Key type ._llcrypt registered [ 3801.931610] Key type .llcrypt registered [ 3802.098222] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 3803.557821] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3803.617847] alg: No test for adler32 (adler32-zlib) [ 3804.745572] Lustre: Lustre: Build Version: 2.17.57_105_g2595dd0 [ 3805.024260] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 3806.740559] Key type lgssc registered [ 3808.145862] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3816.347251] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3816.383616] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3817.809760] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3817.830944] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3817.843662] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.5/24@tcp [ 3822.353658] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3827.169824] LustreError: 127463:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3827.201736] LustreError: 127463:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3831.734077] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3832.046405] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3832.076604] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.5/24@tcp [ 3832.161929] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3836.816433] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3844.547059] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3850.227491] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3857.521146] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3857.924189] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3862.040928] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 3864.842750] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3873.598221] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3887.587478] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3887.602327] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3887.605516] Lustre: Skipped 1 previous similar message [ 3887.622013] Lustre: Skipped 1 previous similar message [ 3889.671180] Lustre: server umount lustre-OST0000 complete [ 3897.828631] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3897.831456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3897.840480] Lustre: Skipped 1 previous similar message [ 3900.309972] Lustre: server umount lustre-MDT0000 complete [ 3902.956108] LustreError: 127463:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3902.986450] LustreError: 127463:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3904.553561] LustreError: 127444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789268856 with bad export cookie 15665069873435172103 [ 3904.569772] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3904.572221] LustreError: 127444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3904.787460] Lustre: server umount lustre-MDT0001 complete [ 3916.058293] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 3919.227644] Key type lgssc unregistered [ 3919.705913] LNet: 131051:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3919.731024] LNetError: 131051:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3919.768413] LNet: Removed LNI 192.168.203.105@tcp [ 3920.801393] Key type .llcrypt unregistered [ 3920.804976] Key type ._llcrypt unregistered [ 3953.240438] Key type ._llcrypt registered [ 3953.244254] Key type .llcrypt registered [ 3953.420157] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_hostid [ 3972.167165] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 3973.139726] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3973.440062] alg: No test for adler32 (adler32-zlib) [ 3974.426749] Lustre: Lustre: Build Version: 2.17.57_105_g2595dd0 [ 3974.577179] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 3976.239370] Key type lgssc registered [ 3978.195722] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4015.819818] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4015.857933] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4017.142300] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4017.182958] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4017.284632] Lustre: lustre-MDT0000: new disk, initializing [ 4017.358429] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4017.380784] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4021.794882] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4031.493061] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4031.635212] Lustre: 134449:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4031.693291] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4031.698210] Lustre: Skipped 1 previous similar message [ 4031.805728] Lustre: lustre-MDT0001: new disk, initializing [ 4031.888453] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4031.949701] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4031.964557] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4036.326357] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4044.884049] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4050.441304] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4057.013620] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4057.319565] Lustre: lustre-OST0000: new disk, initializing [ 4057.325567] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4057.339936] Lustre: 136416:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4057.468134] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4063.432378] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4066.880943] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4066.895996] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4066.981026] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4072.931580] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4077.565451] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4077.733495] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4081.793062] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4081.967562] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4087.778340] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4087.780380] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4087.794019] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4087.805145] Lustre: Skipped 1 previous similar message [ 4092.903495] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4092.912230] Lustre: Skipped 1 previous similar message [ 4098.015128] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4098.016369] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4098.043753] Lustre: Skipped 1 previous similar message [ 4098.248530] Lustre: server umount lustre-OST0000 complete [ 4103.141956] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4103.147228] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4103.160279] Lustre: Skipped 2 previous similar messages [ 4108.058746] Lustre: server umount lustre-MDT0000 complete [ 4112.014886] LustreError: 134441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789269064 with bad export cookie 5666673930716201969 [ 4112.018334] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4112.026059] LustreError: 134441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4112.310408] Lustre: server umount lustre-MDT0001 complete [ 4120.834457] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4121.355046] LustreError: 138635:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4121.365481] LustreError: 138635:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4121.434665] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4126.236184] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4126.688649] LustreError: 138636:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4131.812239] LustreError: 138635:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4135.815659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4136.268423] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4140.306424] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4147.867679] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4153.332322] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4158.837402] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4159.216941] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4164.764920] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4171.428191] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4176.300716] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4176.545272] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4181.265105] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4181.544637] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4189.289721] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4189.522890] Lustre: lustre-OST0001: new disk, initializing [ 4189.528572] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4189.543773] Lustre: 142006:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4189.650474] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4195.473669] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4195.487634] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4195.619991] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4195.673212] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4203.326291] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4203.517247] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4208.070477] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4208.229420] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4213.665612] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff952f8891f800.ost_server_uuid 50 [ 4215.529590] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff952f8891f800.ost_server_uuid in FULL state after 0 sec [ 4225.288207] Lustre: server umount lustre-OST0001 complete [ 4226.537890] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4226.541642] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4226.556173] LustreError: 140458:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4226.574278] LustreError: 140458:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4231.656538] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4231.665835] LustreError: 143472:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4231.666555] Lustre: Skipped 1 previous similar message [ 4231.671066] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4231.671074] Lustre: Skipped 1 previous similar message [ 4231.682088] LustreError: 143472:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4234.605962] Lustre: server umount lustre-OST0000 complete [ 4238.816155] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4238.827372] LustreError: Skipped 1 previous similar message [ 4238.830752] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4241.890924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4241.903593] Lustre: Skipped 3 previous similar messages [ 4243.832695] Lustre: server umount lustre-MDT0000 complete [ 4246.533190] LustreError: 138616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789269198 with bad export cookie 5666673930716202851 [ 4246.534544] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4246.546689] LustreError: 138616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4246.803659] Lustre: server umount lustre-MDT0001 complete [ 4263.227262] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 4265.558886] Key type lgssc unregistered [ 4265.794491] LNet: 145010:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4265.804454] LNetError: 145010:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4265.831265] LNet: Removed LNI 192.168.203.105@tcp [ 4266.469213] Key type .llcrypt unregistered [ 4266.471842] Key type ._llcrypt unregistered [ 4305.104934] Key type ._llcrypt registered [ 4305.106701] Key type .llcrypt registered [ 4305.177540] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_hostid [ 4318.819677] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 4319.787100] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4319.892452] alg: No test for adler32 (adler32-zlib) [ 4320.885614] Lustre: Lustre: Build Version: 2.17.57_105_g2595dd0 [ 4321.099410] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 4322.727160] Key type lgssc registered [ 4323.828262] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4352.590045] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4352.603930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4353.846604] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4353.874450] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4353.966485] Lustre: lustre-MDT0000: new disk, initializing [ 4354.064156] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4354.084970] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4358.958896] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4368.127398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4368.204259] Lustre: 149159:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4368.292267] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4368.296769] Lustre: Skipped 1 previous similar message [ 4368.365779] Lustre: lustre-MDT0001: new disk, initializing [ 4368.466103] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4368.506868] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4368.513481] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4373.343486] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4383.461091] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4389.116317] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4395.665317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4395.894786] Lustre: lustre-OST0000: new disk, initializing [ 4395.898520] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4395.905473] Lustre: 151126:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4395.983499] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4401.074360] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4404.333150] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4404.352696] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4404.529075] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4413.274918] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4418.211609] Lustre: Failing over lustre-OST0000 [ 4418.661633] Lustre: server umount lustre-OST0000 complete [ 4436.447405] Lustre: 147383:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789269372/real 1789269372] req@000000006b509b7a x1876184817238656/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789269388 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4436.448747] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4436.471152] Lustre: 147383:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4436.485962] Lustre: Skipped 1 previous similar message [ 4439.238089] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4439.614735] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4439.630705] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4441.631948] Lustre: 147382:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789269377/real 1789269377] req@00000000a1edda21 x1876184817240448/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789269393 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4442.656558] Lustre: 147383:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789269377/real 1789269377] req@00000000c80dc958 x1876184817240576/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789269393 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4446.105420] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4446.241152] Lustre: 147383:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789269382/real 1789269382] req@00000000d9bad62c x1876184817241600/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789269398 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4446.786395] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4446.802327] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4446.804078] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4450.441036] Lustre: Failing over lustre-MDT0000 [ 4450.704053] Lustre: server umount lustre-MDT0000 complete [ 4451.487383] Lustre: 147384:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789269387/real 1789269387] req@00000000d6521d11 x1876184817242752/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789269403 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4451.527303] Lustre: 147384:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4451.827729] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4456.342589] LustreError: 149165:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4456.364912] LustreError: 149165:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 4456.929954] LustreError: 149170:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4456.956237] LustreError: 149170:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4461.460592] LustreError: 149165:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4466.575071] LustreError: 149166:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4466.591647] LustreError: 149166:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4467.168067] Lustre: 147384:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789269403/real 1789269403] req@000000007c81cf3e x1876184817252608/t0(0) o400->MGC192.168.203.105@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789269419 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4467.203935] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4469.963499] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4471.631071] LustreError: 149165:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4471.658261] LustreError: 149165:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4477.420633] LustreError: 147380:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000004ff6ca63 x1876184817258112/t0(0) o250->MGC192.168.203.105@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4477.713523] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4477.761617] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4477.784027] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4482.010606] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4483.067936] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4483.075160] Lustre: Skipped 1 previous similar message [ 4483.095509] Lustre: lustre-MDT0000: Recovery over after 0:06, of 2 clients 2 recovered and 0 were evicted. [ 4483.126775] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4493.282202] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4493.283888] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4493.286614] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4493.301345] Lustre: Skipped 3 previous similar messages [ 4496.600548] Lustre: server umount lustre-OST0000 complete [ 4503.524330] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4503.529819] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4503.541967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4503.541974] Lustre: Skipped 1 previous similar message [ 4503.560900] Lustre: Skipped 1 previous similar message [ 4506.927876] Lustre: server umount lustre-MDT0000 complete [ 4508.644129] LustreError: 153673:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4508.674686] LustreError: 153673:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 4510.940441] LustreError: 149149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789269463 with bad export cookie 16009795464833563682 [ 4510.943444] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4510.955580] LustreError: 149149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4511.397515] Lustre: server umount lustre-MDT0001 complete [ 4522.859574] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 4525.694782] Key type lgssc unregistered [ 4525.980071] LNet: 154781:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4525.988295] LNetError: 154781:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4526.001880] LNet: Removed LNI 192.168.203.105@tcp [ 4526.916426] Key type .llcrypt unregistered [ 4526.918992] Key type ._llcrypt unregistered [ 4564.153946] Key type ._llcrypt registered [ 4564.157517] Key type .llcrypt registered [ 4564.300727] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 4565.240533] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4565.337975] alg: No test for adler32 (adler32-zlib) [ 4566.441739] Lustre: Lustre: Build Version: 2.17.57_105_g2595dd0 [ 4566.717146] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 4568.463335] Key type lgssc registered [ 4569.353207] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4577.027775] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4577.044463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4578.481481] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4582.914923] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4588.005327] LustreError: 156299:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4588.028082] LustreError: 156299:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4591.019776] LustreError: 156298:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4591.766745] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4595.228787] LustreError: 156327:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4595.242973] LustreError: 156327:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4595.252384] LustreError: 156327:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 17, retries 0, failed: rc = -5 [ 4595.559715] Lustre: server umount lustre-MDT0000 complete [ 4611.236252] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_hostid [ 4624.030056] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 4664.225105] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4664.502898] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4664.542349] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4664.661285] Lustre: lustre-MDT0000: new disk, initializing [ 4664.763303] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4664.780515] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4670.136863] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4681.950428] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4682.091194] Lustre: 159629:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4682.150538] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4682.154758] Lustre: Skipped 1 previous similar message [ 4682.255939] Lustre: lustre-MDT0001: new disk, initializing [ 4682.375860] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4682.425160] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4682.437417] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4686.905111] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4698.357837] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4706.686943] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4715.550210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4716.029096] Lustre: lustre-OST0000: new disk, initializing [ 4716.031898] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4716.049822] Lustre: 161599:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4716.235220] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4721.784968] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4721.792680] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4721.949708] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4723.725424] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4734.087546] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4738.261863] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4740.303317] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 23:21:30 (1789269690) [ 5009.897537] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5009.909858] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5009.923440] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5013.994384] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5014.009909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5014.014280] Lustre: Skipped 1 previous similar message [ 5014.039413] Lustre: Skipped 2 previous similar messages [ 5017.749648] Lustre: server umount lustre-MDT0000 complete [ 5019.119383] LustreError: 159639:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5019.143719] LustreError: 159639:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5021.841420] LustreError: 159622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789269973 with bad export cookie 13014581895201068409 [ 5021.845165] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5021.848897] LustreError: 159622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5024.224697] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5024.231106] LustreError: 159637:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5024.248609] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5026.272748] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5028.460546] Lustre: server umount lustre-MDT0001 complete [ 5048.288640] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5048.543533] Lustre: server umount lustre-OST0000 complete [ 5058.913537] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_hostid [ 5070.174354] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 5107.527477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5107.817954] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5107.857677] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5107.948715] Lustre: lustre-MDT0000: new disk, initializing [ 5108.097512] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5108.113330] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5113.057720] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 5123.680771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5123.795891] Lustre: 166268:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5123.872238] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5123.878821] Lustre: Skipped 1 previous similar message [ 5123.995355] Lustre: lustre-MDT0001: new disk, initializing [ 5124.128429] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5124.200491] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5124.221737] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5128.730514] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 5139.383129] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5145.340589] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5152.475405] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5152.754647] Lustre: lustre-OST0000: new disk, initializing [ 5152.759493] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5152.774795] Lustre: 168240:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5152.866128] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5159.876711] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 5162.067804] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5162.081751] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5162.160113] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5170.788141] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5176.138847] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5176.471806] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5181.925974] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5182.120066] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5188.070598] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5188.076498] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5188.089602] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5188.099722] Lustre: Skipped 1 previous similar message [ 5188.108692] LustreError: Skipped 1 previous similar message [ 5198.309282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5198.316910] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 4. Is it stuck? [ 5198.327560] Lustre: Skipped 4 previous similar messages [ 5198.591657] Lustre: server umount lustre-OST0000 complete [ 5203.424983] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5203.449664] Lustre: Skipped 1 previous similar message [ 5209.494715] Lustre: server umount lustre-MDT0000 complete [ 5213.667525] LustreError: 166278:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5213.685762] LustreError: 166278:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5214.151901] LustreError: 167066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789270166 with bad export cookie 13014581895201227239 [ 5214.166406] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5214.783865] Lustre: server umount lustre-MDT0001 complete [ 5224.493558] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 23:29:34 (1789270174) [ 5242.602683] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_hostid [ 5251.456366] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 5289.997085] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5290.280452] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5290.323337] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5290.447322] Lustre: lustre-MDT0000: new disk, initializing [ 5290.546334] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5290.594298] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5295.643591] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 5307.267452] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5307.427106] Lustre: 173007:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5307.526757] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 5307.531884] Lustre: Skipped 4 previous similar messages [ 5307.690916] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5312.813111] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 5323.117298] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5330.395271] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5337.180494] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5337.476171] Lustre: lustre-OST0000: new disk, initializing [ 5337.482160] Lustre: Skipped 1 previous similar message [ 5337.485911] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5337.488992] Lustre: Skipped 2 previous similar messages [ 5337.501835] Lustre: 174985:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5344.855900] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5344.870822] Lustre: Skipped 1 previous similar message [ 5344.886810] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5344.896141] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 5345.035788] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5354.982950] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5365.736455] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5365.743317] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5365.749048] LustreError: Skipped 1 previous similar message [ 5365.750026] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5365.791790] Lustre: Skipped 1 previous similar message [ 5370.123649] Lustre: server umount lustre-OST0000 complete [ 5380.911820] Lustre: server umount lustre-MDT0000 complete [ 5381.090229] LustreError: 173019:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5381.111581] LustreError: 173019:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5384.670643] LustreError: 173001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789270336 with bad export cookie 13014581895201228121 [ 5384.680669] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5384.689232] LustreError: 173001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 5384.993271] Lustre: server umount lustre-MDT0001 complete [ 5393.765117] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 23:32:23 (1789270343) [ 5411.394406] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_hostid [ 5420.856588] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 5457.032352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5457.293144] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5457.327417] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5457.415721] Lustre: lustre-MDT0000: new disk, initializing [ 5457.497428] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5457.504464] Lustre: Skipped 2 previous similar messages [ 5457.518803] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5461.679466] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 5472.600162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5472.800496] Lustre: 179311:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5473.120152] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5477.234923] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 5485.904897] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5490.920152] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5496.912658] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5497.137928] Lustre: 181283:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5502.662506] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5502.717640] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 5511.352192] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5518.314586] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5518.325331] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5518.332957] Lustre: Skipped 2 previous similar messages [ 5518.342352] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5518.350331] Lustre: Skipped 4 previous similar messages [ 5524.130587] Lustre: server umount lustre-OST0000 complete [ 5533.504886] Lustre: server umount lustre-MDT0000 complete [ 5533.680069] LustreError: 179321:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5533.703405] LustreError: 179321:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5536.956766] LustreError: 182391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789270489 with bad export cookie 13014581895201229206 [ 5536.960630] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5536.975603] LustreError: 182391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5537.267325] Lustre: server umount lustre-MDT0001 complete [ 5547.798735] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 5550.351632] Key type lgssc unregistered [ 5550.632795] LNet: 183141:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5550.640921] LNetError: 183141:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5550.659815] LNet: Removed LNI 192.168.203.105@tcp [ 5551.564499] Key type .llcrypt unregistered [ 5551.569260] Key type ._llcrypt unregistered [ 5575.935516] Key type ._llcrypt registered [ 5575.938393] Key type .llcrypt registered [ 5576.089602] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 5577.231051] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5577.300852] alg: No test for adler32 (adler32-zlib) [ 5578.421231] Lustre: Lustre: Build Version: 2.17.57_105_g2595dd0 [ 5578.616679] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 5580.263130] Key type lgssc registered [ 5581.464863] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5589.567242] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5589.577234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5591.059715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5595.743724] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 5601.248563] LustreError: 184552:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5601.279353] LustreError: 184552:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5604.738346] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5605.125202] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5609.582520] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 5617.057653] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5623.055530] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5629.940495] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5630.276835] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5636.068320] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 5636.404186] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5643.150356] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5671.406555] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5671.414688] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5671.430546] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5671.447456] Lustre: Skipped 1 previous similar message [ 5675.930632] Lustre: server umount lustre-OST0000 complete [ 5681.633576] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5681.646322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5681.659811] Lustre: Skipped 1 previous similar message [ 5681.675767] Lustre: Skipped 2 previous similar messages [ 5686.138299] Lustre: server umount lustre-MDT0000 complete [ 5686.753767] LustreError: 184552:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5686.777811] LustreError: 184552:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5689.720799] LustreError: 184533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789270641 with bad export cookie 10431723612543943584 [ 5689.724076] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5689.733038] LustreError: 184533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5690.036787] Lustre: server umount lustre-MDT0001 complete [ 5700.776445] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 5703.317851] Key type lgssc unregistered [ 5703.653668] LNet: 188108:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5703.659469] LNetError: 188108:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5703.672476] LNet: Removed LNI 192.168.203.105@tcp [ 5704.528913] Key type .llcrypt unregistered [ 5704.532760] Key type ._llcrypt unregistered [ 5728.898392] Key type ._llcrypt registered [ 5728.903499] Key type .llcrypt registered [ 5729.016693] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 5730.017154] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5730.090912] alg: No test for adler32 (adler32-zlib) [ 5731.120373] Lustre: Lustre: Build Version: 2.17.57_105_g2595dd0 [ 5731.370503] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 5733.031191] Key type lgssc registered [ 5734.026420] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5741.065624] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5741.087826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5742.536389] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5747.618919] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 5752.801175] LustreError: 189538:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5752.813277] LustreError: 189538:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5756.899094] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5757.389256] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5762.110232] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 5769.656779] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5776.446890] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5783.953350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5784.309755] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5787.392755] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5792.197860] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 5804.565676] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5811.668959] Lustre: lustre-OST0000: Not available for connect from 192.168.203.5@tcp (stopping) [ 5813.225791] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5813.240965] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5813.249837] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5813.256632] LustreError: Skipped 1 previous similar message [ 5813.273198] Lustre: Skipped 1 previous similar message [ 5816.745110] Lustre: lustre-OST0000: Not available for connect from 192.168.203.5@tcp (stopping) [ 5816.761377] Lustre: Skipped 1 previous similar message [ 5817.436405] Lustre: server umount lustre-OST0000 complete [ 5825.241369] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff952f9130e800.ost_server_uuid 50 [ 5826.835540] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5841.320155] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5841.324400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5841.339516] Lustre: Skipped 1 previous similar message [ 5841.357484] Lustre: Skipped 1 previous similar message [ 5846.498087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5846.507182] Lustre: Skipped 1 previous similar message [ 5847.746076] Lustre: server umount lustre-MDT0000 complete [ 5851.494156] LustreError: 189517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789270803 with bad export cookie 11339327835232695280 [ 5851.496842] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5851.502437] LustreError: 189517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5851.939373] Lustre: server umount lustre-MDT0001 complete [ 5861.053325] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 23:40:11 (1789270811) [ 5867.182855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5867.562675] LustreError: 193458:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5867.636233] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5872.608817] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 5873.121320] LustreError: 193459:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5877.219084] LustreError: 193458:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5881.599262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5882.056449] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5886.979503] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 5895.490629] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5902.764091] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5911.856444] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5912.036959] LustreError: 195277:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5912.227810] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5917.692695] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5919.246099] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 5928.213394] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5936.464123] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5936.739156] Lustre: lustre-OST0001: new disk, initializing [ 5936.744363] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5936.756433] Lustre: 196302:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5936.896674] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5944.339986] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 5946.971354] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5946.982789] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5947.099725] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5954.715798] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5958.626568] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5958.644575] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5958.670033] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5962.748213] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5962.753486] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5964.573143] Lustre: server umount lustre-OST0000 complete [ 5965.729207] LustreError: 195277:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5971.392821] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5971.573267] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5976.839407] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5977.140551] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5983.209211] LustreError: 195280:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5983.242644] LustreError: 195280:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 5993.445679] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5993.455204] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5993.460380] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5993.475567] Lustre: Skipped 1 previous similar message [ 5993.500495] Lustre: Skipped 4 previous similar messages [ 5997.733552] Lustre: server umount lustre-OST0001 complete [ 6003.680448] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6003.697925] Lustre: Skipped 1 previous similar message [ 6008.293174] Lustre: server umount lustre-MDT0000 complete [ 6012.438608] LustreError: 193438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789270964 with bad export cookie 11339327835232696351 [ 6012.447766] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6012.460391] LustreError: 193438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6012.843131] Lustre: server umount lustre-MDT0001 complete [ 6040.501994] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 23:43:10 (1789270990) [ 6050.117112] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6050.517638] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6050.567289] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6050.806163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6055.022677] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 6064.865897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6064.928827] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6065.001211] Lustre: 199743:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6065.138675] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6065.146867] Lustre: Skipped 2 previous similar messages [ 6070.545115] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 6077.916437] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6084.667841] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6092.064294] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6092.290578] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6099.422101] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 6101.100515] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6106.963045] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6114.660475] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6114.734249] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6114.967110] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6114.977077] Lustre: Skipped 2 previous similar messages [ 6122.726249] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 6130.607749] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6136.770583] Lustre: lustre-OST0000: Not available for connect from 192.168.203.5@tcp (stopping) [ 6136.807466] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6136.808623] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6136.847486] Lustre: Skipped 1 previous similar message [ 6139.504028] Lustre: server umount lustre-OST0000 complete [ 6141.846291] LustreError: 201594:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6141.862289] LustreError: 201594:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 6156.774574] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6161.281165] Lustre: server umount lustre-OST0001 complete [ 6171.750740] Lustre: server umount lustre-MDT0000 complete [ 6175.555215] LustreError: 200492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789271127 with bad export cookie 11339327835232697758 [ 6175.563749] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6175.572962] LustreError: 200492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6176.157269] Lustre: server umount lustre-MDT0001 complete [ 6201.031239] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 23:45:51 (1789271151) [ 6219.188760] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_hostid [ 6228.644697] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 6261.897337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6262.123975] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6262.156535] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6262.231838] Lustre: lustre-MDT0000: new disk, initializing [ 6262.330287] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6262.380885] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6267.057160] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 6277.483038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6277.626986] Lustre: 207836:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6277.702480] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6277.709716] Lustre: Skipped 1 previous similar message [ 6277.818857] Lustre: lustre-MDT0001: new disk, initializing [ 6277.988415] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6278.008364] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6282.937562] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 6294.895319] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6302.814617] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6309.876387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6310.177573] Lustre: lustre-OST0000: new disk, initializing [ 6310.181377] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6310.188259] Lustre: 209806:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6316.381995] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 6319.190152] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6319.209109] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6319.268188] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6325.967859] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6330.462304] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6330.672754] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6335.418294] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6335.626288] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6340.064404] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6340.067287] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6340.071348] LustreError: Skipped 2 previous similar messages [ 6340.076575] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6340.092891] Lustre: Skipped 5 previous similar messages [ 6340.094803] Lustre: Skipped 7 previous similar messages [ 6343.500140] Lustre: server umount lustre-OST0000 complete [ 6353.419738] Lustre: server umount lustre-MDT0000 complete [ 6355.423843] LustreError: 207849:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6355.433661] LustreError: 207849:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 6356.908054] LustreError: 207830:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789271309 with bad export cookie 11339327835232699319 [ 6356.912894] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6356.917404] LustreError: 207830:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6357.168543] Lustre: server umount lustre-MDT0001 complete [ 6364.430509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6369.087779] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 6376.841365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6381.654717] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 6389.075813] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6394.755712] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6401.287216] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6406.876904] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 6413.593541] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6417.639322] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6417.829388] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6421.776754] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6422.004726] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6426.085025] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6426.103937] Lustre: Skipped 2 previous similar messages [ 6426.110431] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6426.122049] LustreError: Skipped 1 previous similar message [ 6437.855123] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6438.082389] Lustre: server umount lustre-OST0000 complete [ 6444.416572] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6448.798075] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6477.341024] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6483.141290] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 6489.972729] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6491.051982] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6491.066355] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6491.067925] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6496.743730] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 5 sec [ 6501.344483] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6501.359730] LustreError: Skipped 1 previous similar message [ 6506.679602] Lustre: server umount lustre-OST0000 complete [ 6517.094188] Lustre: server umount lustre-MDT0000 complete [ 6521.517802] LustreError: 213869:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789271473 with bad export cookie 11339327835232700201 [ 6521.525264] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6521.546852] LustreError: 213869:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6521.897872] Lustre: server umount lustre-MDT0001 complete [ 6532.728846] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 23:51:22 (1789271482) [ 6539.297326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6539.714667] LustreError: 217991:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6539.729156] LustreError: 217991:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6539.776632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6539.779368] Lustre: Skipped 6 previous similar messages [ 6543.811783] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 6551.472777] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6555.980176] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 6562.726690] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6567.318606] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6573.256207] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6578.941599] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 6585.622593] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6589.406783] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6589.592080] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6593.418139] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6593.587274] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6599.385329] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6599.571310] Lustre: lustre-OST0001: new disk, initializing [ 6599.583178] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6599.593611] Lustre: 221360:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6605.099125] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 6607.955066] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6607.967739] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6608.057595] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6613.964354] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6618.761761] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6618.993769] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6623.632189] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6623.887269] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6628.841901] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6628.843637] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6628.854555] Lustre: Skipped 6 previous similar messages [ 6628.870388] Lustre: Skipped 20 previous similar messages [ 6632.290071] Lustre: server umount lustre-OST0001 complete [ 6639.112045] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6639.333961] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6643.868812] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6644.110533] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6672.314655] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6677.480947] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6677.486855] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6677.495353] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6677.507791] Lustre: Skipped 2 previous similar messages [ 6678.096827] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 6684.394424] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6686.133759] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6692.833081] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6692.839226] LustreError: Skipped 1 previous similar message [ 6694.149959] Lustre: server umount lustre-OST0001 complete [ 6716.152176] Lustre: server umount lustre-OST0000 complete [ 6726.363972] Lustre: server umount lustre-MDT0000 complete [ 6729.718833] LustreError: 217972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789271681 with bad export cookie 11339327835232701167 [ 6729.720108] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6729.738126] LustreError: 217972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6736.225845] Lustre: server umount lustre-MDT0001 complete [ 6762.390309] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 23:55:12 (1789271712) [ 6771.377700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6771.628178] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6771.655409] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6776.713293] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 6784.862168] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6784.989642] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6785.024630] Lustre: 226606:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6789.623729] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 6796.999859] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6802.849966] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6809.680576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6809.965157] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6816.998224] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 6824.382569] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6833.591362] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6833.699880] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6840.627472] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 6848.578756] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6850.267479] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6853.871781] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6854.046709] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6858.664639] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6858.861300] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6865.666313] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff952f87b52800.ost_server_uuid 50 [ 6867.778953] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff952f87b52800.ost_server_uuid in IDLE state after 0 sec [ 6869.290750] Lustre: Permanently deactivating lustre-OST0001 [ 6869.332374] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6874.432342] Lustre: Permanently reactivating lustre-OST0001 [ 6874.451327] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6874.458705] Lustre: Skipped 2 previous similar messages [ 6886.882548] Lustre: server umount lustre-OST0001 complete [ 6887.408606] LustreError: 228453:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6887.446321] LustreError: 228453:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 36 previous similar messages [ 6892.520810] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6892.536309] Lustre: Skipped 9 previous similar messages [ 6897.853715] Lustre: server umount lustre-OST0000 complete [ 6907.756305] Lustre: server umount lustre-MDT0000 complete [ 6911.500546] LustreError: 226598:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789271863 with bad export cookie 11339327835232702448 [ 6911.514121] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6911.524447] LustreError: 226598:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6911.735820] Lustre: server umount lustre-MDT0001 complete [ 6934.896642] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 23:58:05 (1789271885) [ 6942.222500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6942.468482] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6942.486396] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6942.493438] Lustre: Skipped 2 previous similar messages [ 6946.700335] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 6954.676265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6954.791206] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6954.819292] Lustre: 233304:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6959.587754] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 6967.448716] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6971.955361] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6978.158702] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6978.374128] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6978.394140] Lustre: Permanently deactivating lustre-OST0000 [ 6978.404490] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6984.558657] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 6987.844934] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6992.369601] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6999.260314] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6999.397427] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7005.856470] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 7009.000733] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7013.687418] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7015.807422] Lustre: Permanently reactivating lustre-OST0000 [ 7015.826299] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7015.831327] Lustre: Skipped 2 previous similar messages [ 7018.902486] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7018.906081] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 31 seconds. I think it's dead, and I am evicting it. exp 000000002fd43a0c, cur 1789271971 deadline 1789271970 last 1789271940 [ 7018.910803] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7018.986632] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7019.080189] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7034.849140] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7034.857089] LustreError: Skipped 2 previous similar messages [ 7039.164991] Lustre: server umount lustre-OST0001 complete [ 7049.520571] Lustre: server umount lustre-OST0000 complete [ 7060.976394] Lustre: server umount lustre-MDT0000 complete [ 7064.963782] LustreError: 233295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789272017 with bad export cookie 11339327835232704128 [ 7064.982698] LustreError: 233295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7064.985339] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7065.461142] Lustre: server umount lustre-MDT0001 complete [ 7075.233422] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 7077.577207] Key type lgssc unregistered [ 7077.789678] LNet: 238187:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7077.797309] LNetError: 238187:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7077.817558] LNet: Removed LNI 192.168.203.105@tcp [ 7078.503508] Key type .llcrypt unregistered [ 7078.506613] Key type ._llcrypt unregistered [ 7105.118860] Key type ._llcrypt registered [ 7105.122753] Key type .llcrypt registered [ 7105.308590] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 7106.666150] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7106.906279] alg: No test for adler32 (adler32-zlib) [ 7108.123258] Lustre: Lustre: Build Version: 2.17.57_105_g2595dd0 [ 7108.318760] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 7110.018357] Key type lgssc registered [ 7111.844471] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7126.912407] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7126.981852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7128.598176] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7136.391121] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 7138.788900] LustreError: 239664:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7138.855450] LustreError: 239664:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7143.917856] LustreError: 239665:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7147.604057] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7147.784898] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7147.841052] Lustre: 239654:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7147.861700] Lustre: Permanently deactivating lustre-MDT0001 [ 7147.895625] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7148.173330] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7152.329778] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 7154.208448] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7159.987266] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7165.407405] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7171.976612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7172.383724] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7173.427253] LustreError: 241506:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7173.471596] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7173.483967] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7178.733914] LustreError: 241586:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7178.757344] LustreError: 241586:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7178.887400] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 7183.847784] LustreError: 241506:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7183.876687] LustreError: 241506:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7186.378840] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7193.017449] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7193.298479] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7198.724611] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7198.738318] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7199.464256] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 7207.103899] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7210.341521] Lustre: Permanently reactivating lustre-MDT0001 [ 7210.353022] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7210.359460] Lustre: Skipped 2 previous similar messages [ 7215.021611] Lustre: lustre-MDT0001: Client e7826e7c-13d8-446a-b459-d8a88f366692 (at 192.168.203.5@tcp) reconnecting [ 7220.419739] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7220.438603] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7220.452016] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7220.470951] LustreError: 243403:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 7220.473282] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7220.481810] LustreError: 243403:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 7220.481921] LustreError: 243403:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.105@tcp: cfg command failed: rc = -108 [ 7220.481943] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 7220.481943] [ 7220.488185] LustreError: 239656:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 7220.563531] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7222.198647] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7228.462227] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7228.843636] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7234.382572] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7234.654652] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7237.439248] Lustre: Permanently deactivating lustre-MDT0001 [ 7237.452102] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7237.455989] Lustre: Skipped 2 previous similar messages [ 7247.050657] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7257.056712] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7257.065063] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7257.073636] Lustre: Skipped 3 previous similar messages [ 7257.078447] Lustre: Skipped 2 previous similar messages [ 7262.180674] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7262.192081] Lustre: Skipped 4 previous similar messages [ 7262.836714] Lustre: server umount lustre-MDT0000 complete [ 7266.597840] LustreError: 239646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789272218 with bad export cookie 12379316360879102564 [ 7266.605401] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7266.612799] LustreError: 239646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7266.895923] Lustre: server umount lustre-MDT0001 complete [ 7272.265319] Lustre: server umount lustre-OST0000 complete [ 7283.056879] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 00:03:53 (1789272233) [ 7295.412864] Lustre: server umount lustre-OST0001 complete [ 7304.060102] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_hostid [ 7315.713748] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 7353.340725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7353.610614] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7353.621375] Lustre: Skipped 2 previous similar messages [ 7353.637296] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7353.697336] Lustre: lustre-MDT0000: new disk, initializing [ 7353.751142] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7353.767432] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7358.239545] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 7368.990748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7369.108756] Lustre: 247975:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7369.150928] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7369.158433] Lustre: Skipped 1 previous similar message [ 7369.216303] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7369.245185] Lustre: lustre-MDT0001: new disk, initializing [ 7369.321299] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7369.359994] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7369.369078] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7374.217277] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 7382.708787] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7388.391386] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7395.800420] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7396.048691] Lustre: lustre-OST0000: new disk, initializing [ 7396.056248] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7396.066479] Lustre: 249941:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7396.162744] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7402.365496] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 7405.149610] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7405.160036] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7405.282507] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7411.772440] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7414.225120] LustreError: 247981:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7423.146046] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7423.290828] Lustre: lustre-OST0001: new disk, initializing [ 7423.295410] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7423.299840] Lustre: 251057:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7424.240073] LustreError: 247981:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7424.269454] LustreError: 250710:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7428.619432] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 7431.726566] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7431.739963] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7431.813833] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7434.367072] LustreError: 250710:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7436.894454] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7438.314418] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7441.889686] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7441.891993] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7441.904977] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7441.942850] Lustre: Skipped 1 previous similar message [ 7446.385841] Lustre: server umount lustre-OST0001 complete [ 7447.013017] LustreError: 249923:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7447.040560] LustreError: 249923:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7466.976758] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7466.977123] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7466.989507] LustreError: Skipped 1 previous similar message [ 7466.991207] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7467.003219] Lustre: Skipped 1 previous similar message [ 7467.024186] Lustre: Skipped 3 previous similar messages [ 7467.508131] LustreError: 249926:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7467.525348] LustreError: 249926:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 7477.740897] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7477.747077] Lustre: Skipped 5 previous similar messages [ 7478.239136] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7478.334546] Lustre: server umount lustre-OST0000 complete [ 7481.831481] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7481.839746] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7496.161835] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7496.633079] Lustre: server umount lustre-MDT0000 complete [ 7502.726485] LustreError: 247967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789272454 with bad export cookie 12379316360879105371 [ 7502.746121] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7502.753462] LustreError: 247967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7517.151362] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7517.380662] Lustre: server umount lustre-MDT0001 complete [ 7530.001324] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 7533.442280] Key type lgssc unregistered [ 7533.928647] LNet: 253097:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7533.957704] LNetError: 253097:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7533.985320] LNet: Removed LNI 192.168.203.105@tcp [ 7535.366572] Key type .llcrypt unregistered [ 7535.369673] Key type ._llcrypt unregistered [ 7578.510608] Key type ._llcrypt registered [ 7578.512669] Key type .llcrypt registered [ 7578.621742] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 7579.970731] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7580.019480] alg: No test for adler32 (adler32-zlib) [ 7581.060507] Lustre: Lustre: Build Version: 2.17.57_105_g2595dd0 [ 7581.303743] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 7582.967173] Key type lgssc registered [ 7584.080182] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7592.086947] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7592.107147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7593.414600] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7593.471760] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7593.674827] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7597.870603] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 7606.319897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7606.374876] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7606.398432] Lustre: 255302:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7606.639537] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7610.700280] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 7617.904323] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7624.098722] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7635.395846] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7635.519818] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7635.655294] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7643.022921] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 7644.865467] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7652.583101] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7706.592385] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7706.600888] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7706.615061] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7706.630172] LustreError: Skipped 1 previous similar message [ 7706.643149] Lustre: Skipped 1 previous similar message [ 7711.429502] Lustre: server umount lustre-OST0000 complete [ 7715.136816] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7726.325216] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7726.591749] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7732.391120] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 7740.009660] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7746.029867] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7746.045409] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7746.046289] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7746.061721] Lustre: Skipped 1 previous similar message [ 7781.864229] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7781.866823] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7781.909411] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7781.913452] Lustre: Skipped 2 previous similar messages [ 7783.128588] Lustre: server umount lustre-OST0000 complete [ 7791.072872] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7791.079720] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7791.096106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7791.099723] Lustre: Skipped 1 previous similar message [ 7797.222412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7797.240723] Lustre: Skipped 3 previous similar messages [ 7802.335670] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7802.341208] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7802.380063] Lustre: Skipped 1 previous similar message [ 7802.729059] Lustre: server umount lustre-MDT0000 complete [ 7807.377174] LustreError: 255295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789272759 with bad export cookie 14553482235891015499 [ 7807.381936] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7807.384025] LustreError: 255295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7807.762298] Lustre: server umount lustre-MDT0001 complete [ 7820.615990] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 7823.670559] Key type lgssc unregistered [ 7823.970221] LNet: 260730:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7823.983060] LNetError: 260730:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7825.005250] LNet: Removed LNI 192.168.203.105@tcp [ 7825.691310] Key type .llcrypt unregistered [ 7825.693246] Key type ._llcrypt unregistered [ 7855.458908] Key type ._llcrypt registered [ 7855.471486] Key type .llcrypt registered [ 7855.658522] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 7856.738470] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7856.895572] alg: No test for adler32 (adler32-zlib) [ 7858.186818] Lustre: Lustre: Build Version: 2.17.57_105_g2595dd0 [ 7858.522170] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 7860.199168] Key type lgssc registered [ 7861.720623] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7870.740824] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7870.751418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7872.247168] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7876.044434] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 7882.722416] LustreError: 262159:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7882.743900] LustreError: 262159:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7885.416987] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7885.809188] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7891.277864] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 7900.148815] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7909.123383] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7919.882834] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7920.723588] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7930.447640] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 7940.712884] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7942.349701] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7973.355893] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7973.359313] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7973.398843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7973.437805] Lustre: Skipped 1 previous similar message [ 7975.570591] Lustre: server umount lustre-OST0000 complete [ 7983.073371] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7983.089958] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7983.098917] Lustre: Skipped 1 previous similar message [ 7983.108249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7985.833150] Lustre: server umount lustre-MDT0000 complete [ 7988.709948] LustreError: 262159:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7988.725934] LustreError: 262159:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7989.168671] LustreError: 262137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789272941 with bad export cookie 9396686228624473360 [ 7989.173661] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7989.187613] LustreError: 262137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7989.444421] Lustre: server umount lustre-MDT0001 complete [ 8002.321766] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 8004.936374] Key type lgssc unregistered [ 8005.153491] LNet: 266466:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8005.156426] LNetError: 266466:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8006.181815] LNet: Removed LNI 192.168.203.105@tcp [ 8006.904867] Key type .llcrypt unregistered [ 8006.907156] Key type ._llcrypt unregistered [ 8027.308169] Key type ._llcrypt registered [ 8027.318900] Key type .llcrypt registered [ 8027.476457] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 8029.142808] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8029.167281] alg: No test for adler32 (adler32-zlib) [ 8030.355173] Lustre: Lustre: Build Version: 2.17.57_105_g2595dd0 [ 8030.612810] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 8032.401476] Key type lgssc registered [ 8033.444815] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8041.098663] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8041.114504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8042.607847] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8047.030687] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 8052.195661] LustreError: 267686:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8052.210309] LustreError: 267686:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8056.388168] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8056.939952] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8060.661718] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 8067.462057] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8073.493717] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8082.131827] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8082.538924] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8087.686894] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8089.298050] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 8099.017339] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8113.632593] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8113.636099] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8113.643640] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8116.705163] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8116.705943] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8116.711363] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8116.721036] Lustre: Skipped 1 previous similar message [ 8119.118848] Lustre: server umount lustre-OST0000 complete [ 8123.362786] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8123.368582] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8123.380350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8126.945263] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8126.951075] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8129.517125] Lustre: server umount lustre-MDT0000 complete [ 8132.073515] LustreError: 268781:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8132.093942] LustreError: 268781:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8132.894217] LustreError: 267667:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789273085 with bad export cookie 974374173212938908 [ 8132.899099] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8132.915819] LustreError: 267667:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8133.220521] Lustre: server umount lustre-MDT0001 complete [ 8143.565667] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 8146.968677] Key type lgssc unregistered [ 8147.545624] LNet: 271362:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8147.565623] LNetError: 271362:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8147.593532] LNet: Removed LNI 192.168.203.105@tcp [ 8148.897186] Key type .llcrypt unregistered [ 8148.899866] Key type ._llcrypt unregistered [ 8178.264725] Key type ._llcrypt registered [ 8178.267474] Key type .llcrypt registered [ 8178.455818] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 8179.940907] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8180.019016] alg: No test for adler32 (adler32-zlib) [ 8181.180776] Lustre: Lustre: Build Version: 2.17.57_105_g2595dd0 [ 8181.435036] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 8183.071888] Key type lgssc registered [ 8184.173740] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8192.523982] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8192.549318] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8194.142718] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8199.502672] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 8204.261533] LustreError: 272777:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8204.303989] LustreError: 272777:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8208.822103] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8209.337808] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8213.388198] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 8220.262832] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8226.097835] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8231.797235] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8232.183095] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8236.575891] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 8239.347166] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8243.042859] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8270.313813] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8270.315387] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8270.330850] LustreError: Skipped 1 previous similar message [ 8270.337856] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8270.353636] Lustre: Skipped 1 previous similar message [ 8270.390603] Lustre: Skipped 1 previous similar message [ 8271.789195] Lustre: server umount lustre-OST0000 complete [ 8275.945296] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8275.952361] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8275.980952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8280.546323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8280.547317] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8285.665273] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8285.669790] Lustre: Skipped 2 previous similar messages [ 8290.271622] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8290.490198] Lustre: server umount lustre-MDT0000 complete [ 8290.785660] LustreError: 272774:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8290.804986] LustreError: 272774:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8295.434098] LustreError: 272757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789273247 with bad export cookie 283372533842362805 [ 8295.439926] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8295.456806] LustreError: 272757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8295.944959] Lustre: server umount lustre-MDT0001 complete [ 8309.779982] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 8313.057291] Key type lgssc unregistered [ 8313.443508] LNet: 277083:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8313.451889] LNetError: 277083:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8313.475423] LNet: Removed LNI 192.168.203.105@tcp [ 8314.309309] Key type .llcrypt unregistered [ 8314.312053] Key type ._llcrypt unregistered [ 8335.036245] Key type ._llcrypt registered [ 8335.040828] Key type .llcrypt registered [ 8335.183627] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 8336.203141] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8336.240406] alg: No test for adler32 (adler32-zlib) [ 8337.392547] Lustre: Lustre: Build Version: 2.17.57_105_g2595dd0 [ 8337.795099] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 8339.458759] Key type lgssc registered [ 8340.688399] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8348.455917] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8348.482446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8349.991700] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8354.434050] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 8360.426560] LustreError: 278318:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8360.446831] LustreError: 278318:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8363.055274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8363.522667] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8368.256907] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 8376.418879] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8381.829679] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8389.839863] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8390.298967] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8395.830608] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8399.933968] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 8412.338789] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8426.464091] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8426.472047] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8426.497459] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8428.512715] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8428.522022] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8430.691417] Lustre: server umount lustre-OST0000 complete [ 8435.169460] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8435.176805] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8435.196935] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8435.200187] Lustre: Skipped 1 previous similar message [ 8438.754891] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8438.757097] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8441.331441] Lustre: server umount lustre-MDT0000 complete [ 8443.872271] LustreError: 278319:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8443.883081] LustreError: 278319:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8446.058262] LustreError: 278283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789273398 with bad export cookie 17041018658303638598 [ 8446.064289] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8446.092285] LustreError: 278283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8446.671419] Lustre: server umount lustre-MDT0001 complete [ 8461.309332] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 8466.069301] Key type lgssc unregistered [ 8466.408152] LNet: 282019:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8466.412810] LNetError: 282019:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8467.429857] LNet: Removed LNI 192.168.203.105@tcp [ 8468.666687] Key type .llcrypt unregistered [ 8468.669898] Key type ._llcrypt unregistered [ 8483.460614] Key type ._llcrypt registered [ 8483.462579] Key type .llcrypt registered [ 8483.607742] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8500.885589] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_hostid [ 8520.793255] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 8521.761361] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8521.848197] alg: No test for adler32 (adler32-zlib) [ 8523.024178] Lustre: Lustre: Build Version: 2.17.57_105_g2595dd0 [ 8523.302453] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 8525.103503] Key type lgssc registered [ 8526.897816] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8567.976757] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8568.000633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8569.251753] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8569.327978] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8569.466434] Lustre: lustre-MDT0000: new disk, initializing [ 8569.572736] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8569.621853] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8576.195617] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 8588.354584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8588.507391] Lustre: 285572:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8588.592208] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8588.599530] Lustre: Skipped 1 previous similar message [ 8588.701209] Lustre: lustre-MDT0001: new disk, initializing [ 8588.825496] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8588.877613] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8588.896898] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8592.307733] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 8603.182863] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8611.093946] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8619.480970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8619.940747] Lustre: lustre-OST0000: new disk, initializing [ 8619.946034] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8619.960592] Lustre: 287541:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8620.087957] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8625.725846] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8625.739841] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8625.887327] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8628.052509] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 8640.007690] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8646.739704] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8647.039878] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8652.777268] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8653.001395] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8656.865325] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8656.879739] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8656.886058] LustreError: Skipped 1 previous similar message [ 8656.890053] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8656.921073] Lustre: Skipped 1 previous similar message [ 8656.944215] Lustre: Skipped 1 previous similar message [ 8662.265366] Lustre: server umount lustre-OST0000 complete [ 8667.111589] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8667.140333] Lustre: Skipped 1 previous similar message [ 8667.141675] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8667.154502] Lustre: Skipped 1 previous similar message [ 8672.226964] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8672.248290] Lustre: Skipped 1 previous similar message [ 8673.015282] Lustre: server umount lustre-MDT0000 complete [ 8677.218277] LustreError: 285562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789273629 with bad export cookie 14496045495595974627 [ 8677.231123] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8677.240486] LustreError: 285562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8677.632835] Lustre: server umount lustre-MDT0001 complete [ 8688.477767] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 00:27:18 (1789273638) [ 8695.894571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8696.636341] LustreError: 289956:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8696.673359] LustreError: 289956:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8696.798680] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8701.931082] LustreError: 289957:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8702.035361] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 8706.016843] LustreError: 289956:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8709.998712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8710.202968] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8715.214777] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 8725.852522] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8731.987382] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8740.299658] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8740.911614] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8749.347914] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 8758.782940] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8770.534666] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8770.542763] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8770.550059] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8770.571826] Lustre: Skipped 1 previous similar message [ 8775.660976] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8775.668327] Lustre: Skipped 1 previous similar message [ 8776.133682] Lustre: server umount lustre-OST0000 complete [ 8780.772987] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8785.895327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8785.915746] Lustre: Skipped 4 previous similar messages [ 8786.830277] Lustre: server umount lustre-MDT0000 complete [ 8791.011043] LustreError: 289957:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8791.030970] LustreError: 289957:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8791.132723] LustreError: 289936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789273743 with bad export cookie 14496045495595975509 [ 8791.154717] LustreError: 289936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8791.163328] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8791.545396] Lustre: server umount lustre-MDT0001 complete [ 8802.661857] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 8805.409047] Key type lgssc unregistered [ 8805.686134] LNet: 293491:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8805.710722] LNetError: 293491:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8805.732192] LNet: Removed LNI 192.168.203.105@tcp [ 8806.573121] Key type .llcrypt unregistered [ 8806.575437] Key type ._llcrypt unregistered [ 8833.834729] Key type ._llcrypt registered [ 8833.841558] Key type .llcrypt registered [ 8833.979570] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_hostid [ 8851.170087] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 8852.095106] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8852.165921] alg: No test for adler32 (adler32-zlib) [ 8853.212825] Lustre: Lustre: Build Version: 2.17.57_105_g2595dd0 [ 8853.445409] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 8855.207151] Key type lgssc registered [ 8856.173368] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8912.450842] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8912.481895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8913.783318] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8913.869580] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8914.053336] Lustre: lustre-MDT0000: new disk, initializing [ 8914.202311] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8914.219654] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8919.194641] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 8930.242801] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8930.343633] Lustre: 297314:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8930.455509] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8930.459935] Lustre: Skipped 1 previous similar message [ 8930.575951] Lustre: lustre-MDT0001: new disk, initializing [ 8930.750263] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8930.816826] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8930.845182] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8936.646389] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 8946.288222] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8952.025545] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8958.120157] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8958.380931] Lustre: lustre-OST2710: new disk, initializing [ 8958.383917] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8958.386858] Lustre: 299283:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8958.487769] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8963.943768] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 8967.784812] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8967.814469] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8967.927754] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8972.132384] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8978.364321] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8978.574401] Lustre: lustre-OST03e8: new disk, initializing [ 8978.585700] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8978.601440] Lustre: 300346:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8978.729802] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8984.173611] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8984.192844] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8984.322478] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8984.752163] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 8991.651677] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8999.417222] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8999.702725] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9006.360099] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9006.698897] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9013.454448] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9013.780225] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9020.004992] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9020.215607] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9025.506731] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9025.510949] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9025.527986] Lustre: Skipped 1 previous similar message [ 9027.497530] Lustre: server umount lustre-MDT0000 complete [ 9031.126364] LustreError: 297307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789273983 with bad export cookie 9634499222662350076 [ 9031.130144] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9031.149071] LustreError: 297307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9031.476665] Lustre: server umount lustre-MDT0001 complete [ 9045.197053] Lustre: server umount lustre-OST2710 complete [ 9058.258758] Lustre: server umount lustre-OST03e8 complete [ 9079.788698] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_hostid [ 9090.060287] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 9136.409646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9136.713243] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9136.744892] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9136.832900] Lustre: lustre-MDT0000: new disk, initializing [ 9137.008498] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9137.019629] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9142.831431] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 9154.398871] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9154.545752] Lustre: 305597:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9154.640291] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9154.646233] Lustre: Skipped 1 previous similar message [ 9154.764364] Lustre: lustre-MDT0001: new disk, initializing [ 9154.888679] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9154.932794] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9154.947119] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9159.634672] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 9169.593557] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9177.026768] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9185.154100] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9185.515148] Lustre: lustre-OST0000: new disk, initializing [ 9185.519474] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9185.525545] Lustre: 307569:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9185.656620] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9191.525620] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9191.535372] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9191.644803] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9193.016792] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 9203.182381] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9208.958332] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9209.236575] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9215.147967] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9215.311246] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9222.625508] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9222.630474] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9222.644802] LustreError: Skipped 1 previous similar message [ 9222.653627] Lustre: Skipped 2 previous similar messages [ 9222.660438] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9222.660446] Lustre: Skipped 3 previous similar messages [ 9224.910062] Lustre: server umount lustre-OST0000 complete [ 9231.847959] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9231.851607] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9231.888815] Lustre: Skipped 1 previous similar message [ 9231.915835] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9231.928350] Lustre: Skipped 1 previous similar message [ 9236.808374] Lustre: server umount lustre-MDT0000 complete [ 9238.012225] LustreError: 305604:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9238.038607] LustreError: 305604:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 9242.174764] LustreError: 307782:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9242.214578] LustreError: 307782:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9243.374094] LustreError: 306395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789274195 with bad export cookie 9634499222662351756 [ 9243.381187] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9243.404140] LustreError: 306395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9244.415606] Lustre: server umount lustre-MDT0001 complete [ 9255.186435] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 00:36:45 (1789274205) [ 9257.228547] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9259.045167] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 00:36:49 (1789274209) [ 9262.681123] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 9291.941071] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9292.226344] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9292.267261] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9292.464644] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9297.103176] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 9307.278646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9307.399651] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9307.457621] Lustre: 311431:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9307.645870] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 9307.648763] Lustre: Skipped 2 previous similar messages [ 9313.938450] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 9338.175024] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9338.503834] Lustre: Denying initial registration attempt from nid 192.168.203.105@tcp, specified as failover [ 9338.526159] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9338.549085] LustreError: 312885:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -99 [ 9338.566617] LustreError: 312885:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 9338.584036] LustreError: 312885:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9338.750595] Lustre: server umount lustre-OST0000 complete [ 9338.759535] LustreError: 312885:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9349.101528] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9349.115469] Lustre: Skipped 1 previous similar message [ 9349.119675] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9349.121837] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9349.133911] Lustre: Skipped 1 previous similar message [ 9359.331870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9359.346845] Lustre: Skipped 3 previous similar messages [ 9362.399238] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9362.850773] Lustre: server umount lustre-MDT0000 complete [ 9364.529439] LustreError: 311438:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9364.565673] LustreError: 311438:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9369.588036] LustreError: 312887:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9370.139687] LustreError: 311422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789274322 with bad export cookie 9634499222662352638 [ 9370.170795] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9370.173541] LustreError: 311422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9371.049758] Lustre: server umount lustre-MDT0001 complete [ 9386.229486] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 9388.744199] Key type lgssc unregistered [ 9389.035781] LNet: 314008:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9389.044852] LNetError: 314008:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9389.059969] LNet: Removed LNI 192.168.203.105@tcp [ 9389.719178] Key type .llcrypt unregistered [ 9389.723142] Key type ._llcrypt unregistered [ 9400.283421] Key type ._llcrypt registered [ 9400.286027] Key type .llcrypt registered [ 9400.417733] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 9401.258471] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9401.296276] alg: No test for adler32 (adler32-zlib) [ 9402.386415] Lustre: Lustre: Build Version: 2.17.57_105_g2595dd0 [ 9402.582309] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 9404.239290] Key type lgssc registered [ 9405.272870] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9458.672214] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 9470.253908] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9470.266422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9471.744694] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9471.829332] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9472.046385] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9476.956537] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 9487.896701] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9488.015548] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9488.084394] Lustre: 316770:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9488.318738] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9492.771050] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 9500.877123] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9507.271027] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9528.171907] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9528.345498] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9528.528740] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9532.964453] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 9539.750558] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9545.696901] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9545.705419] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9545.736895] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9546.211329] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9546.242846] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9546.255468] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9546.258212] Lustre: Skipped 1 previous similar message [ 9548.399975] Lustre: server umount lustre-OST0000 complete [ 9554.916084] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9554.931326] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9554.981518] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9560.027401] Lustre: server umount lustre-MDT0000 complete [ 9563.902549] LustreError: 317511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789274516 with bad export cookie 8988858370140552574 [ 9563.907869] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9563.914483] LustreError: 317511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9564.209267] Lustre: server umount lustre-MDT0001 complete [ 9576.508605] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 9579.318541] Key type lgssc unregistered [ 9579.634828] LNet: 320341:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9579.645151] LNetError: 320341:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9579.669075] LNet: Removed LNI 192.168.203.105@tcp [ 9580.428699] Key type .llcrypt unregistered [ 9580.431655] Key type ._llcrypt unregistered [ 9605.326286] Key type ._llcrypt registered [ 9605.328907] Key type .llcrypt registered [ 9605.436338] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 9606.720490] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9606.751165] alg: No test for adler32 (adler32-zlib) [ 9607.930510] Lustre: Lustre: Build Version: 2.17.57_105_g2595dd0 [ 9608.283096] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 9610.047205] Key type lgssc registered [ 9611.116706] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9618.301206] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9618.343207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9619.737152] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9624.214228] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 9629.155846] LustreError: 321753:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9629.178323] LustreError: 321753:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9633.540865] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9633.838755] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9637.285681] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 9645.104463] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9654.805139] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9665.829432] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9666.051272] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9672.796300] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 9682.178589] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9704.383486] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.5@tcp (stopping) [ 9704.930200] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9704.941366] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9704.970792] Lustre: Skipped 2 previous similar messages [ 9704.994949] Lustre: Skipped 2 previous similar messages [ 9705.660943] Lustre: server umount lustre-MDT0000 complete [ 9709.460425] LustreError: 321749:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9709.881859] LustreError: 323602:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789274662 with bad export cookie 10402454194142830204 [ 9709.903548] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9709.913303] LustreError: 323602:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9710.054977] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9710.070169] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9714.080370] LustreError: 321750:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9714.084265] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9714.117055] LustreError: 321750:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9719.265614] LustreError: 324072:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9719.265665] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9719.289729] LustreError: 324072:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9719.309309] Lustre: Skipped 1 previous similar message [ 9724.383141] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9724.800626] Lustre: server umount lustre-MDT0001 complete [ 9731.539927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9741.981092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9755.109422] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x905cf0f99a27fbda [ 9755.118292] Lustre: MGC192.168.203.105@tcp: Connection restored to 0@lo (at 0@lo) [ 9755.405813] LustreError: 323590:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9755.420811] LustreError: 323590:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9755.494045] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9755.563943] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9759.739267] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 9765.857476] LustreError: 323590:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9765.892235] LustreError: 323590:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9770.614759] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9770.979922] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 9770.993943] Lustre: Skipped 2 previous similar messages [ 9771.152331] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9775.294658] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9776.670732] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 9788.594627] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9790.681888] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9797.193368] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9799.478478] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9806.308233] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9806.313904] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9806.321704] Lustre: Skipped 1 previous similar message [ 9811.302581] Lustre: server umount lustre-OST0000 complete [ 9816.551588] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9816.579239] Lustre: Skipped 1 previous similar message [ 9821.994585] Lustre: server umount lustre-MDT0000 complete [ 9826.061118] LustreError: 321734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789274778 with bad export cookie 10402454194142837722 [ 9826.063315] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9826.069978] LustreError: 321734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9826.336921] Lustre: server umount lustre-MDT0001 complete [ 9839.196670] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 9842.216994] Key type lgssc unregistered [ 9842.568890] LNet: 328053:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9842.583918] LNetError: 328053:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9842.641783] LNet: Removed LNI 192.168.203.105@tcp [ 9843.915573] Key type .llcrypt unregistered [ 9843.918259] Key type ._llcrypt unregistered [ 9871.532457] Key type ._llcrypt registered [ 9871.535140] Key type .llcrypt registered [ 9871.667376] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 9873.060790] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9873.157057] alg: No test for adler32 (adler32-zlib) [ 9874.133563] Lustre: Lustre: Build Version: 2.17.57_105_g2595dd0 [ 9874.412810] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 9876.087165] Key type lgssc registered [ 9877.103750] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9884.791723] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9884.822408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9886.274196] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9891.076111] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 9896.424313] LustreError: 329468:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9896.450735] LustreError: 329468:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9901.538891] LustreError: 329467:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9901.950980] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9902.418071] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9908.134563] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 9918.784695] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9926.209397] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9932.260983] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9932.265325] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9932.291979] Lustre: Skipped 1 previous similar message [ 9932.307579] Lustre: Skipped 1 previous similar message [ 9936.571427] Lustre: server umount lustre-MDT0000 complete [ 9937.376544] LustreError: 329462:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9937.406618] LustreError: 329462:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9940.159401] LustreError: 329446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789274892 with bad export cookie 13998250350080098792 [ 9940.165056] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9940.171772] LustreError: 329446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9940.448089] Lustre: server umount lustre-MDT0001 complete [ 9954.479985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9954.907314] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9954.954981] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9955.296089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9961.069397] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 9969.829807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9969.938831] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9970.002086] Lustre: 332055:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9970.329916] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9975.765970] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 9982.752742] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9988.632382] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9999.465746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9999.651671] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9999.666515] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9999.681405] LustreError: 333940:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -2 [ 9999.689694] LustreError: 333940:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 9999.694743] LustreError: 333940:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9999.750838] Lustre: server umount lustre-OST0000 complete [ 9999.754265] LustreError: 333940:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10010.995936] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10011.263114] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10011.578084] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10018.858444] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [10021.163898] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10027.121407] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10034.330974] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10034.457299] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10034.572878] Lustre: lustre-OST0001: new disk, initializing [10034.579513] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10034.591652] Lustre: 335483:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10034.741047] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10040.818647] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [10041.919888] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10041.924602] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10041.989162] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10048.077923] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10052.578695] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10052.583409] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10052.587727] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10052.604279] Lustre: Skipped 1 previous similar message [10057.697716] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10057.708947] Lustre: Skipped 2 previous similar messages [10062.827139] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10062.834930] Lustre: Skipped 1 previous similar message [10065.377040] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10065.465663] Lustre: server umount lustre-OST0001 complete [10067.949015] LustreError: 334407:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10067.972320] LustreError: 334407:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10072.548973] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10072.549672] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10072.557169] LustreError: Skipped 2 previous similar messages [10072.561685] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10072.570848] Lustre: Skipped 1 previous similar message [10072.593077] Lustre: Skipped 1 previous similar message [10073.060098] LustreError: 334408:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10073.102352] LustreError: 334408:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10075.327475] Lustre: server umount lustre-OST0000 complete [10082.785499] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10082.802830] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10082.829778] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10082.841174] Lustre: Skipped 2 previous similar messages [10085.554348] Lustre: server umount lustre-MDT0000 complete [10088.426462] LustreError: 332063:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10088.455979] LustreError: 332063:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10089.938154] LustreError: 332049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789275042 with bad export cookie 13998250350080099345 [10089.951921] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10089.964808] LustreError: 332049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10090.492944] Lustre: server umount lustre-MDT0001 complete [10105.897413] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [10108.308960] Key type lgssc unregistered [10108.523869] LNet: 337445:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10108.528912] LNetError: 337445:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10108.543660] LNet: Removed LNI 192.168.203.105@tcp [10109.645995] Key type .llcrypt unregistered [10109.648386] Key type ._llcrypt unregistered [10183.357794] Key type ._llcrypt registered [10183.360663] Key type .llcrypt registered [10183.498606] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_hostid [10198.520709] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [10199.575753] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10199.600456] alg: No test for adler32 (adler32-zlib) [10200.687391] Lustre: Lustre: Build Version: 2.17.57_105_g2595dd0 [10200.962639] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [10202.591537] Key type lgssc registered [10203.761516] Lustre: Echo OBD driver; http://www.lustre.org/ [10242.494398] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10242.526493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10243.816421] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10243.854497] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10243.929511] Lustre: lustre-MDT0000: new disk, initializing [10244.004616] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10244.029589] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10248.226852] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [10259.218233] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10259.433381] Lustre: 342376:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10259.590882] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10259.604766] Lustre: Skipped 1 previous similar message [10259.837076] Lustre: lustre-MDT0001: new disk, initializing [10260.020144] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10260.102951] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10260.118739] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10264.849570] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [10277.085659] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10283.706675] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10292.708432] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10293.115366] Lustre: lustre-OST0000: new disk, initializing [10293.124301] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10293.137503] Lustre: 344344:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10293.288139] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10299.678865] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [10300.996309] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10301.017316] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10301.241996] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10309.766530] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10314.496175] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10314.739983] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10320.790172] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10321.077154] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10327.011181] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10327.023870] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10327.032126] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10332.170126] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10332.174205] Lustre: Skipped 1 previous similar message [10337.249189] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10337.257399] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10337.277967] Lustre: Skipped 1 previous similar message [10337.454211] Lustre: server umount lustre-OST0000 complete [10341.857159] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10341.871539] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10341.880273] Lustre: Skipped 1 previous similar message [10341.887707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10341.895059] Lustre: Skipped 1 previous similar message [10347.489710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10347.496696] Lustre: Skipped 3 previous similar messages [10347.828424] Lustre: server umount lustre-MDT0000 complete [10351.619961] LustreError: 344343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789275303 with bad export cookie 12242619935230088462 [10351.625803] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10351.644096] LustreError: 344343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10352.234616] Lustre: server umount lustre-MDT0001 complete [10361.386336] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 00:55:11 (1789275311) [10369.438555] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 00:55:19 (1789275319) [10377.046344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10377.668256] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10381.935166] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [10382.828268] LustreError: 347100:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10382.858471] LustreError: 347100:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10386.915304] LustreError: 347099:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10390.062570] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10390.444077] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10394.953642] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [10402.119235] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10407.161831] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10413.906446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10414.265376] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10420.271300] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [10428.639476] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10436.976365] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10438.658771] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10440.486346] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10442.554837] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10444.908780] Lustre: Failing over lustre-MDT0000 [10445.225943] Lustre: server umount lustre-MDT0000 complete [10446.778903] LustreError: 347096:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10446.819160] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10446.828681] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10446.854545] Lustre: Skipped 1 previous similar message [10451.858647] LustreError: 347096:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10451.883216] LustreError: 347096:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10456.986271] LustreError: 348341:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10457.013293] LustreError: 348341:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10463.865683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10464.103471] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10464.463880] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10464.556663] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10467.214331] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10469.820093] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [10469.894841] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10469.958272] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [10470.015023] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10480.268363] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10482.335735] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10485.943836] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10488.105911] Lustre: Failing over lustre-MDT0000 [10488.566699] Lustre: server umount lustre-MDT0000 complete [10488.754989] LustreError: 347096:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10488.791984] LustreError: 347096:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [10490.339895] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10490.344764] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10490.369621] Lustre: Skipped 4 previous similar messages [10505.698421] LustreError: 347100:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10505.722610] LustreError: 347100:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [10506.719330] Lustre: 340596:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789275442/real 1789275442] req@00000000d00903c3 x1876190982926336/t0(0) o400->MGC192.168.203.105@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789275458 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10506.753481] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10507.578149] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10515.936405] LustreError: 340595:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000a8f641ec x1876190982931968/t0(0) o250->MGC192.168.203.105@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10516.511558] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10516.635910] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10519.454647] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10521.601647] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10521.639946] Lustre: Skipped 2 previous similar messages [10521.674896] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [10521.780777] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10526.070842] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [10539.169915] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10540.682452] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10547.170374] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10547.181390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10547.183137] Lustre: Skipped 2 previous similar messages [10547.196889] Lustre: Skipped 2 previous similar messages [10552.291154] LustreError: 347099:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10552.312555] LustreError: 347099:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [10552.385496] Lustre: server umount lustre-MDT0000 complete [10556.148918] LustreError: 347079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789275508 with bad export cookie 12242619935230091178 [10556.165529] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10556.507811] Lustre: server umount lustre-MDT0001 complete [10571.448729] Lustre: server umount lustre-OST0000 complete [10586.088247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10586.647822] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10590.997785] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [10599.620422] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10604.271938] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [10611.134779] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10616.768960] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10623.725211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10630.994792] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [10631.342388] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10638.907710] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10645.691696] Lustre: server umount lustre-OST0000 complete [10651.104981] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10651.113659] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10651.127922] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10659.404368] Lustre: server umount lustre-MDT0000 complete [10662.367418] Lustre: 340596:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789275598/real 1789275598] req@000000001e7c5e2b x1876190982980736/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789275614 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10662.370886] LustreError: 353438:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10662.399891] LustreError: 353438:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10663.078880] LustreError: 353422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789275615 with bad export cookie 12242619935230091920 [10663.083633] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10663.093628] LustreError: 353422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [10663.496728] Lustre: server umount lustre-MDT0001 complete [10674.144816] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [10676.618889] Key type lgssc unregistered [10676.965655] LNet: 356973:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10676.976538] LNetError: 356973:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10677.003207] LNet: Removed LNI 192.168.203.105@tcp [10677.829854] Key type .llcrypt unregistered [10677.832246] Key type ._llcrypt unregistered [10707.715707] Key type ._llcrypt registered [10707.717955] Key type .llcrypt registered [10707.834948] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [10709.134472] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10709.184365] alg: No test for adler32 (adler32-zlib) [10710.170541] Lustre: Lustre: Build Version: 2.17.57_105_g2595dd0 [10710.387891] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [10712.183175] Key type lgssc registered [10713.417941] Lustre: Echo OBD driver; http://www.lustre.org/ [10722.146783] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10722.166904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10723.722859] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10728.845568] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [10733.036776] LustreError: 358500:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10733.060574] LustreError: 358500:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10738.149929] LustreError: 358501:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10738.346518] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10738.766499] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10742.985655] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [10751.043520] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10758.032283] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10765.009506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10765.387546] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10768.495283] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10771.044447] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [10778.540757] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10793.011562] Lustre: server umount lustre-MDT0000 complete [10793.443866] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10793.448096] LustreError: 360998:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10793.461848] Lustre: Skipped 2 previous similar messages [10793.475622] LustreError: 360998:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10797.194355] LustreError: 358482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789275749 with bad export cookie 9346712136222492119 [10797.200470] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10797.218355] LustreError: 358482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10797.674870] Lustre: server umount lustre-MDT0001 complete [10812.534467] Lustre: server umount lustre-OST0000 complete [10826.942711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10827.460201] LustreError: 362235:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10827.575532] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10832.204773] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [10832.869094] LustreError: 362236:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10841.371273] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10841.692329] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10846.395450] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [10854.726747] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10860.797311] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10867.348689] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10867.635678] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10872.743919] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10873.212825] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [10882.450753] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10896.367520] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10896.369858] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10896.400845] Lustre: Skipped 2 previous similar messages [10896.408527] Lustre: Skipped 2 previous similar messages [10901.032845] Lustre: server umount lustre-MDT0000 complete [10901.479650] LustreError: 364777:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10901.491325] LustreError: 364777:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10904.581629] LustreError: 362215:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789275856 with bad export cookie 9346712136222493232 [10904.586322] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10904.598813] LustreError: 362215:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10904.943718] Lustre: server umount lustre-MDT0001 complete [10919.619350] Lustre: server umount lustre-OST0000 complete [10935.616744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10936.040688] LustreError: 366149:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10936.137569] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10941.049703] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [10950.288448] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10955.473837] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [10964.611901] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10971.640781] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10979.822704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10980.114336] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10980.121655] Lustre: Skipped 1 previous similar message [10981.195953] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10986.994382] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [10995.858402] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11001.828354] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11001.831541] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11001.834668] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11001.845347] Lustre: Skipped 1 previous similar message [11003.833939] Lustre: server umount lustre-OST0000 complete [11012.072730] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11012.076775] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11012.082109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11012.082116] Lustre: Skipped 1 previous similar message [11012.130991] Lustre: Skipped 1 previous similar message [11017.193230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11017.200268] Lustre: Skipped 2 previous similar messages [11022.303820] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11022.314939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11022.342558] Lustre: Skipped 1 previous similar message [11022.585153] Lustre: server umount lustre-MDT0000 complete [11026.355153] LustreError: 366129:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789275978 with bad export cookie 9346712136222494254 [11026.376504] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11026.734839] Lustre: server umount lustre-MDT0001 complete [11037.518197] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [11040.255513] Key type lgssc unregistered [11040.518948] LNet: 369682:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11040.528814] LNetError: 369682:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11041.574239] LNet: Removed LNI 192.168.203.105@tcp [11042.492766] Key type .llcrypt unregistered [11042.497939] Key type ._llcrypt unregistered [11069.402681] Key type ._llcrypt registered [11069.405534] Key type .llcrypt registered [11069.520186] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [11070.346184] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11070.484420] alg: No test for adler32 (adler32-zlib) [11071.519178] Lustre: Lustre: Build Version: 2.17.57_105_g2595dd0 [11071.904918] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [11073.623145] Key type lgssc registered [11074.740885] Lustre: Echo OBD driver; http://www.lustre.org/ [11082.492575] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11082.519599] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11082.530143] LustreError: 371119:0:(osd_handler.c:8547:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11082.565546] LustreError: 371119:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11082.575473] LustreError: 371119:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11082.601081] LustreError: 371119:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11082.622160] LustreError: 371119:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11090.854060] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11090.865318] LustreError: 371517:0:(osd_handler.c:8547:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11090.890796] LustreError: 371517:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11090.897175] LustreError: 371517:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11090.905140] LustreError: 371517:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11090.912910] LustreError: 371517:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11109.989405] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [11112.916709] Key type lgssc unregistered [11113.187974] LNet: 372520:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11113.193627] LNetError: 372520:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11113.213254] LNet: Removed LNI 192.168.203.105@tcp [11114.022627] Key type .llcrypt unregistered [11114.026717] Key type ._llcrypt unregistered [11130.989514] Key type ._llcrypt registered [11130.991763] Key type .llcrypt registered [11131.091970] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_hostid [11145.659544] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [11146.711066] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11146.761132] alg: No test for adler32 (adler32-zlib) [11147.818409] Lustre: Lustre: Build Version: 2.17.57_105_g2595dd0 [11148.042671] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [11149.703180] Key type lgssc registered [11150.585238] Lustre: Echo OBD driver; http://www.lustre.org/ [11189.058989] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11189.079206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11190.333757] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11190.398690] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11190.531871] Lustre: lustre-MDT0000: new disk, initializing [11190.682919] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11190.721580] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11195.147597] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [11205.003179] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11205.138074] Lustre: 375672:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11205.206709] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11205.212321] Lustre: Skipped 1 previous similar message [11205.297824] Lustre: lustre-MDT0001: new disk, initializing [11205.406871] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11205.439445] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11205.458402] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11209.975231] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [11219.643766] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11224.711220] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11230.749951] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11230.945406] Lustre: lustre-OST0000: new disk, initializing [11230.948580] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11230.953628] Lustre: 377641:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11231.021549] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11237.063525] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [11240.519276] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11240.536363] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11240.645745] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11247.829533] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11252.683104] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11252.886042] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11258.098523] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11258.422746] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11261.425926] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11261.438396] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11261.465188] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11261.480672] Lustre: Skipped 1 previous similar message [11266.532073] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11266.539899] Lustre: Skipped 1 previous similar message [11266.866507] Lustre: server umount lustre-OST0000 complete [11271.651825] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11271.671620] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11271.671765] Lustre: Skipped 2 previous similar messages [11271.684223] Lustre: Skipped 1 previous similar message [11276.769983] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11276.782540] Lustre: Skipped 1 previous similar message [11277.072657] Lustre: server umount lustre-MDT0000 complete [11280.998280] LustreError: 375665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789276233 with bad export cookie 11511204014488154149 [11281.014252] LustreError: 375665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11281.014826] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11281.369753] Lustre: server umount lustre-MDT0001 complete [11290.349533] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11292.432891] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 01:10:42 (1789276242) [11299.552063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11300.119993] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11305.080964] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [11305.449121] LustreError: 380101:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11305.474061] LustreError: 380101:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11310.561371] LustreError: 380102:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11314.076281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11314.411805] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11318.641651] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [11326.664592] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11332.493984] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11339.750561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11340.149917] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11347.533306] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [11355.635847] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11362.353343] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11362.530140] Lustre: lustre-OST0001: new disk, initializing [11362.531844] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11362.536522] Lustre: 382938:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11362.611191] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11369.858070] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [11371.115289] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11371.132334] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11371.211363] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11379.430562] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11386.798663] Lustre: lustre-OST0001: Not available for connect from 192.168.203.5@tcp (stopping) [11386.849987] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11386.867909] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11386.875344] LustreError: Skipped 2 previous similar messages [11389.738704] Lustre: server umount lustre-OST0001 complete [11391.892814] LustreError: 381922:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11397.099369] LustreError: 381924:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11397.129460] LustreError: 381924:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11402.224761] LustreError: 383367:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11402.254312] LustreError: 383367:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11406.819749] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11406.847110] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11406.871475] Lustre: Skipped 1 previous similar message [11406.876125] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11406.880872] Lustre: Skipped 2 previous similar messages [11412.452696] LustreError: 381924:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11412.477292] LustreError: 381924:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11412.938877] Lustre: server umount lustre-OST0000 complete [11417.593737] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11417.622894] Lustre: Skipped 1 previous similar message [11422.180062] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11423.826940] Lustre: server umount lustre-MDT0000 complete [11428.184426] LustreError: 380081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789276380 with bad export cookie 11511204014488155024 [11428.192942] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11428.195912] LustreError: 380081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11434.868932] Lustre: server umount lustre-MDT0001 complete [11446.811673] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [11450.122220] Key type lgssc unregistered [11450.449366] LNet: 384924:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11450.459911] LNetError: 384924:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11450.474769] LNet: Removed LNI 192.168.203.105@tcp [11451.451751] Key type .llcrypt unregistered [11451.458184] Key type ._llcrypt unregistered [11505.121564] Key type ._llcrypt registered [11505.124561] Key type .llcrypt registered [11505.245451] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [11506.294907] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11506.390063] alg: No test for adler32 (adler32-zlib) [11507.557696] Lustre: Lustre: Build Version: 2.17.57_105_g2595dd0 [11507.900337] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [11509.631578] Key type lgssc registered [11510.810425] Lustre: Echo OBD driver; http://www.lustre.org/ [11518.581424] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11518.600325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11519.922210] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11519.944894] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11520.096442] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11524.117629] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [11532.539921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11532.660227] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11532.693217] Lustre: 387722:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11532.962889] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11537.534371] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [11544.129327] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11550.149766] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11553.249519] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11553.254307] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11553.271424] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11556.323299] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11556.324120] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11558.667349] Lustre: server umount lustre-MDT0000 complete [11563.676246] LustreError: 388472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789276515 with bad export cookie 5241831181256058179 [11563.694151] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11563.709715] LustreError: 388472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11564.268278] Lustre: server umount lustre-MDT0001 complete [11575.340356] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 01:15:25 (1789276525) [11582.662552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11583.139573] LustreError: 390233:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11583.160289] LustreError: 390233:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11583.215754] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11587.964951] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [11588.578637] LustreError: 390234:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11593.697712] LustreError: 390233:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11596.952337] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11597.336383] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11601.851288] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [11610.293650] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11616.516626] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11624.093825] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11624.273643] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11624.485429] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11631.523985] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [11640.524652] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11650.003428] Lustre: Permanently deactivating lustre-OST0000 [11650.028259] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11659.278339] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11661.746234] LustreError: 393265:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11661.753265] LustreError: 393265:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11661.759156] LustreError: 393265:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11665.728123] LustreError: 393314:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11665.749388] LustreError: 393314:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11673.688340] Lustre: server umount lustre-OST0000 complete [11676.187399] LustreError: 393563:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11676.198297] LustreError: 393563:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11678.688919] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11678.691463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11678.703964] Lustre: Skipped 1 previous similar message [11678.709837] Lustre: Skipped 2 previous similar messages [11683.810227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11683.816682] Lustre: Skipped 1 previous similar message [11688.119568] Lustre: server umount lustre-MDT0000 complete [11688.930263] LustreError: 390234:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11688.952680] LustreError: 390234:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11692.002152] LustreError: 390213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789276644 with bad export cookie 5241831181256058760 [11692.014414] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11692.018187] LustreError: 390213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11702.412513] Lustre: server umount lustre-MDT0001 complete [11711.584676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11716.926541] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [11719.693773] LustreError: 394727:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11726.345801] Lustre: 395066:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.105@tcp [11728.318306] Lustre: server umount MGS complete [11741.120082] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11741.693388] LustreError: 395790:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11741.716515] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11741.722121] Lustre: Skipped 1 previous similar message [11741.795431] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11746.477529] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [11751.905202] LustreError: 395791:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11751.929145] LustreError: 395791:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11755.369159] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11755.617568] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11760.291664] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [11768.573558] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11774.511815] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11776.333484] Lustre: Permanently reactivating lustre-OST0000 [11776.356665] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11776.362372] Lustre: Skipped 2 previous similar messages [11802.594855] LustreError: 397524:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11810.256813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11810.577191] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11810.582250] Lustre: Skipped 1 previous similar message [11817.453435] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [11825.806867] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11833.319727] Lustre: server umount lustre-OST0000 complete [11837.929176] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11837.939535] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11837.958477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11847.730777] Lustre: server umount lustre-MDT0000 complete [11848.675883] LustreError: 395790:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11848.698491] LustreError: 395790:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11849.119109] Lustre: 387043:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789276785/real 1789276785] req@0000000040ebf9c4 x1876192353419520/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789276801 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11849.142933] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11849.153740] Lustre: Skipped 1 previous similar message [11851.354106] LustreError: 395770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789276803 with bad export cookie 5241831181256059810 [11851.364357] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11851.786928] Lustre: server umount lustre-MDT0001 complete [11862.050300] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [11864.651515] Key type lgssc unregistered [11864.962694] LNet: 400107:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11864.973789] LNetError: 400107:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11865.014724] LNet: Removed LNI 192.168.203.105@tcp [11865.993389] Key type .llcrypt unregistered [11865.996726] Key type ._llcrypt unregistered [11884.529239] Key type ._llcrypt registered [11884.532502] Key type .llcrypt registered [11884.628978] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_hostid [11899.747765] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [11900.802154] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11900.910389] alg: No test for adler32 (adler32-zlib) [11902.048943] Lustre: Lustre: Build Version: 2.17.57_105_g2595dd0 [11902.358371] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [11904.103158] Key type lgssc registered [11905.358884] Lustre: Echo OBD driver; http://www.lustre.org/ [11940.809532] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11940.829813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11942.056281] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11942.118090] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11942.210556] Lustre: lustre-MDT0000: new disk, initializing [11942.331929] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11942.368434] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11946.741590] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [11957.032147] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11957.167903] Lustre: 403259:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11957.253395] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11957.270681] Lustre: Skipped 1 previous similar message [11957.402483] Lustre: lustre-MDT0001: new disk, initializing [11957.536953] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11957.577567] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11957.597557] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11962.627615] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [11972.048744] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11978.366854] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11986.929866] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11987.451787] Lustre: lustre-OST0000: new disk, initializing [11987.461626] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11987.478538] Lustre: 405229:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11987.671128] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11994.932844] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [11996.783864] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11996.815989] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11997.013067] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12005.732540] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12010.280264] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12010.551603] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12015.130888] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12015.420385] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12017.636628] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12017.649479] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12017.653259] Lustre: Skipped 1 previous similar message [12017.653648] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12022.760885] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12022.770053] Lustre: Skipped 2 previous similar messages [12023.540827] Lustre: server umount lustre-OST0000 complete [12027.874036] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12027.878569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12027.888653] Lustre: Skipped 1 previous similar message [12027.893585] Lustre: Skipped 1 previous similar message [12032.992766] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12033.007133] Lustre: Skipped 1 previous similar message [12035.417848] Lustre: server umount lustre-MDT0000 complete [12039.440496] LustreError: 404055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789276991 with bad export cookie 16818464869394888838 [12039.443421] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12039.453609] LustreError: 404055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12039.741791] Lustre: server umount lustre-MDT0001 complete [12048.387620] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12050.254358] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 01:23:20 (1789277000) [12057.351323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12057.790129] LustreError: 407690:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12057.812423] LustreError: 407690:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12057.855333] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12062.739290] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [12063.207242] LustreError: 407691:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12067.297175] LustreError: 407690:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12070.342239] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12070.664945] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12075.352899] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [12085.025919] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12092.055903] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12099.451481] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12099.867134] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12105.650354] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [12113.380575] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12115.112210] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12117.513781] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12125.665150] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12125.677534] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12125.700188] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12128.861923] Lustre: server umount lustre-OST0000 complete [12135.905599] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12135.916407] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12135.923778] Lustre: Skipped 2 previous similar messages [12135.936381] Lustre: Skipped 2 previous similar messages [12138.984661] Lustre: server umount lustre-MDT0000 complete [12141.025459] LustreError: 408697:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12141.052262] LustreError: 408697:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12142.442579] LustreError: 407672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789277094 with bad export cookie 16818464869394889720 [12142.454634] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12142.457969] LustreError: 407672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12142.817728] Lustre: server umount lustre-MDT0001 complete [12153.983522] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [12156.678768] Key type lgssc unregistered [12157.068064] LNet: 411411:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12157.073798] LNetError: 411411:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12157.104199] LNet: Removed LNI 192.168.203.105@tcp [12158.207255] Key type .llcrypt unregistered [12158.214427] Key type ._llcrypt unregistered [12182.839405] Key type ._llcrypt registered [12182.845411] Key type .llcrypt registered [12182.990075] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [12183.297351] Key type .llcrypt unregistered [12183.300275] Key type ._llcrypt unregistered [12200.169426] Key type ._llcrypt registered [12200.172836] Key type .llcrypt registered [12200.271305] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [12201.160576] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12201.182182] alg: No test for adler32 (adler32-zlib) [12202.314629] Lustre: Lustre: Build Version: 2.17.57_105_g2595dd0 [12202.561053] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [12204.247161] Key type lgssc registered [12205.562673] Lustre: Echo OBD driver; http://www.lustre.org/ [12212.970249] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12213.001719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12214.417331] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12218.227426] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [12223.974342] LustreError: 413615:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12224.000503] LustreError: 413615:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12227.526388] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12227.864132] LustreError: 413616:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12227.907751] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12233.185090] LustreError: 413615:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12233.205108] LustreError: 413615:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12234.130266] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [12238.313420] LustreError: 414357:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12238.343291] LustreError: 414357:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12239.358890] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12241.516574] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12244.451942] LustreError: 413615:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12244.482419] LustreError: 413615:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12248.329643] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12248.660442] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12252.367966] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [12259.810312] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12259.821964] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12259.831697] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12261.743409] Lustre: server umount lustre-OST0000 complete [12268.010403] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12268.019846] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12268.042650] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12269.028533] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12269.052932] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12269.067332] Lustre: Skipped 1 previous similar message [12271.604769] Lustre: server umount lustre-MDT0000 complete [12274.146916] LustreError: 413616:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12274.164892] LustreError: 413616:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12275.411259] LustreError: 415357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789277227 with bad export cookie 339858063621782758 [12275.422907] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12275.430065] LustreError: 415357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12285.052545] Lustre: server umount lustre-MDT0001 complete [12296.891795] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [12301.098667] Key type lgssc unregistered [12301.564749] LNet: 416762:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12301.577888] LNetError: 416762:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12301.632932] LNet: Removed LNI 192.168.203.105@tcp [12302.810634] Key type .llcrypt unregistered [12302.812789] Key type ._llcrypt unregistered [12332.471709] Key type ._llcrypt registered [12332.475710] Key type .llcrypt registered [12332.591939] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [12333.488562] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12333.514039] alg: No test for adler32 (adler32-zlib) [12334.456642] Lustre: Lustre: Build Version: 2.17.57_105_g2595dd0 [12334.706470] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [12336.423546] Key type lgssc registered [12337.398418] Lustre: Echo OBD driver; http://www.lustre.org/ [12345.261326] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12345.282925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12360.799415] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12365.919459] LustreError: 418150:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.105@tcp: failed processing log, type 4: rc = -110 [12392.608712] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12399.839928] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [12408.428560] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12415.785072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12416.362484] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12416.434630] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12421.378194] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [12421.608038] LustreError: 418175:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12421.632198] LustreError: 418175:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12426.727158] LustreError: 418176:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12427.755349] LustreError: 418175:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12427.798961] LustreError: 418175:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12431.427510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12431.897728] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12436.645025] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [12444.513728] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12451.217274] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12457.457922] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12457.460631] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12457.464232] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12457.466358] LustreError: Skipped 1 previous similar message [12457.487387] Lustre: Skipped 1 previous similar message [12458.737661] Lustre: server umount lustre-OST0000 complete [12465.125787] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12465.132190] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12465.142176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12465.146579] Lustre: Skipped 1 previous similar message [12467.682914] LustreError: 419192:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12467.683448] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12467.747058] Lustre: server umount lustre-MDT0000 complete [12472.802318] LustreError: 419187:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12472.802944] LustreError: 420678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789277424 with bad export cookie 10974997279013618963 [12472.821933] LustreError: 419187:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12472.823406] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12472.834351] LustreError: 420678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12473.314739] Lustre: server umount lustre-MDT0001 complete [12486.456649] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [12489.927441] Key type lgssc unregistered [12490.251338] LNet: 421717:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12490.260440] LNetError: 421717:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12490.278209] LNet: Removed LNI 192.168.203.105@tcp [12491.376757] Key type .llcrypt unregistered [12491.379302] Key type ._llcrypt unregistered [12520.711434] Key type ._llcrypt registered [12520.722252] Key type .llcrypt registered [12520.978637] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [12521.793669] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12521.810970] alg: No test for adler32 (adler32-zlib) [12522.868557] Lustre: Lustre: Build Version: 2.17.57_105_g2595dd0 [12523.027110] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [12524.753468] Key type lgssc registered [12526.859609] Lustre: Echo OBD driver; http://www.lustre.org/ [12536.042553] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12536.058543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12537.563900] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12542.421516] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [12547.044456] LustreError: 423130:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12547.083146] LustreError: 423130:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12551.191937] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12551.444461] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12556.189798] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [12564.879773] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12573.028849] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12583.922256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12584.462945] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12592.323608] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [12600.086926] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12607.032646] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12617.158737] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.5@tcp (stopping) [12617.189408] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12617.213771] Lustre: Skipped 2 previous similar messages [12620.817392] Lustre: server umount lustre-MDT0000 complete [12622.226155] LustreError: 423124:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12623.666481] LustreError: 423109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.5@tcp arrived at 1789277575 with bad export cookie 15984471420127802963 [12623.687508] LustreError: 423109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12625.798311] LustreError: 423110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789277577 with bad export cookie 15984471420127802298 [12625.803620] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12625.814685] LustreError: 423110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12627.426076] LustreError: 423129:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12627.440982] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12627.441996] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12627.455021] LustreError: 423129:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12627.485244] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12627.499720] Lustre: Skipped 3 previous similar messages [12630.434583] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12630.436201] LustreError: 423125:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12630.470203] LustreError: 423125:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12631.932535] Lustre: server umount lustre-OST0000 complete [12635.558928] LustreError: 423129:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12635.596797] LustreError: 423129:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12637.990259] Lustre: server umount lustre-MDT0001 complete [12649.336366] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [12651.825716] Key type lgssc unregistered [12652.155842] LNet: 426839:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12652.165386] LNetError: 426839:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12652.184116] LNet: Removed LNI 192.168.203.105@tcp [12653.217183] Key type .llcrypt unregistered [12653.220580] Key type ._llcrypt unregistered [12678.198874] Key type ._llcrypt registered [12678.202879] Key type .llcrypt registered [12678.304724] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [12679.470681] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12679.555725] alg: No test for adler32 (adler32-zlib) [12680.625434] Lustre: Lustre: Build Version: 2.17.57_105_g2595dd0 [12680.892438] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [12682.583174] Key type lgssc registered [12683.702861] Lustre: Echo OBD driver; http://www.lustre.org/ [12692.018775] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12692.053536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12693.583789] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12698.311627] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [12703.712783] LustreError: 428253:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12703.733754] LustreError: 428253:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12707.181034] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12707.621265] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12712.026278] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [12719.599388] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12725.915520] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12733.221053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12733.667907] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12738.806838] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12739.436483] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [12746.495203] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12753.303062] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.5@tcp (stopping) [12753.890495] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12753.900939] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12753.915529] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12754.403690] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12756.702379] Lustre: server umount lustre-MDT0001 complete [12758.429470] LustreError: 429688:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12759.521339] LustreError: 428253:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12759.539845] LustreError: 428253:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12764.640566] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12764.644591] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12764.644857] LustreError: 428254:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12764.669391] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12764.713233] Lustre: Skipped 2 previous similar messages [12769.172329] Lustre: server umount lustre-OST0000 complete [12769.763231] LustreError: 429296:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12772.757810] Lustre: server umount lustre-MDT0000 complete [12785.825385] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [12788.787640] Key type lgssc unregistered [12789.031658] LNet: 431936:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12789.039608] LNetError: 431936:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12789.053173] LNet: Removed LNI 192.168.203.105@tcp [12789.891149] Key type .llcrypt unregistered [12789.892906] Key type ._llcrypt unregistered [12817.442327] Key type ._llcrypt registered [12817.444798] Key type .llcrypt registered [12817.534171] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_hostid [12834.086958] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [12835.208162] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12835.260762] alg: No test for adler32 (adler32-zlib) [12836.395097] Lustre: Lustre: Build Version: 2.17.57_105_g2595dd0 [12836.661296] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [12838.407915] Key type lgssc registered [12839.517437] Lustre: Echo OBD driver; http://www.lustre.org/ [12877.528640] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [12893.303710] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12893.364560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12894.540089] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12894.580251] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12894.741252] Lustre: lustre-MDT0000: new disk, initializing [12894.879542] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12894.911547] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12898.776905] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [12917.294644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12917.716034] Lustre: lustre-OST0000: new disk, initializing [12917.722441] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12917.732744] Lustre: Skipped 1 previous similar message [12917.743965] Lustre: 436965:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12917.847647] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12924.593492] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [12926.545592] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12926.553142] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12926.576066] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12933.783300] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12948.963184] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12949.068226] Lustre: 435948:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12949.126344] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12949.180055] Lustre: lustre-MDT0001: new disk, initializing [12949.326366] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12949.366303] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12949.374939] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12952.791804] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [12961.930367] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12962.257750] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12967.371597] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12967.612560] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12972.956769] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.5@tcp (stopping) [12975.089044] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12975.097773] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12975.112995] Lustre: Skipped 1 previous similar message [12975.126040] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12978.068278] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.5@tcp (stopping) [12978.078768] Lustre: Skipped 1 previous similar message [12978.736371] Lustre: server umount lustre-MDT0001 complete [12985.316913] LustreError: 438304:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12985.352930] LustreError: 438304:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [12990.433964] LustreError: 435959:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12990.458044] LustreError: 435959:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12995.558570] LustreError: 435955:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12995.580511] LustreError: 435955:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12998.112545] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12998.125972] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12998.155716] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13000.675041] LustreError: 435959:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13002.400350] Lustre: server umount lustre-OST0000 complete [13011.941436] Lustre: server umount lustre-MDT0000 complete [13023.282964] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [13026.525457] Key type lgssc unregistered [13026.871726] LNet: 440717:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13026.878344] LNetError: 440717:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13026.898651] LNet: Removed LNI 192.168.203.105@tcp [13027.921609] Key type .llcrypt unregistered [13027.927406] Key type ._llcrypt unregistered [13071.128136] Key type ._llcrypt registered [13071.146342] Key type .llcrypt registered [13071.417850] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [13071.942350] Key type .llcrypt unregistered [13071.944202] Key type ._llcrypt unregistered [13090.025495] Key type ._llcrypt registered [13090.029784] Key type .llcrypt registered [13090.135263] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [13091.535595] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13091.614811] alg: No test for adler32 (adler32-zlib) [13092.729438] Lustre: Lustre: Build Version: 2.17.57_105_g2595dd0 [13093.090242] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [13094.839279] Key type lgssc registered [13096.096729] Lustre: Echo OBD driver; http://www.lustre.org/ [13125.168337] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13125.198142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13126.469277] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13126.497196] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13126.587524] Lustre: lustre-MDT0000: new disk, initializing [13126.705932] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13126.730231] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13130.937927] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [13141.979633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13142.132288] Lustre: 444084:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13142.245930] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13142.258270] Lustre: Skipped 1 previous similar message [13142.402358] Lustre: lustre-MDT0001: new disk, initializing [13142.512551] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13142.578857] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13142.598111] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13147.336874] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [13157.573477] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13163.829761] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13170.609630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13170.942969] Lustre: lustre-OST0000: new disk, initializing [13170.945931] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13170.952656] Lustre: 446052:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13171.050464] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13177.421523] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13177.435041] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13177.557761] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13177.674834] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [13187.396530] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13198.308081] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13198.313937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13198.335950] Lustre: Skipped 2 previous similar messages [13198.349432] Lustre: Skipped 2 previous similar messages [13202.425716] Lustre: server umount lustre-MDT0000 complete [13206.248831] LustreError: 444079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789278158 with bad export cookie 978620700138552728 [13206.251395] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13206.265118] LustreError: 444079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13206.765942] Lustre: server umount lustre-MDT0001 complete [13221.518516] Lustre: server umount lustre-OST0000 complete [13232.916292] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 01:43:03 (1789278183) [13240.082187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13240.675157] LustreError: 448082:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13240.702530] LustreError: 448082:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [13240.782991] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13245.920113] LustreError: 448083:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13245.932028] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [13250.019502] LustreError: 448082:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13254.291115] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13254.693634] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13259.554880] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [13266.978396] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13272.696765] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13291.767710] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13292.014741] Lustre: Found index 0 for lustre-OST0000, updating log [13292.210617] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13298.240751] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [13305.794917] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13311.662492] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13322.211963] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13322.214920] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13322.236056] LustreError: Skipped 1 previous similar message [13322.244414] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13322.260134] Lustre: Skipped 1 previous similar message [13323.304802] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13323.316644] Lustre: Skipped 1 previous similar message [13328.280783] Lustre: server umount lustre-OST0000 complete [13346.784857] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13346.796238] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13346.814712] Lustre: Skipped 1 previous similar message [13346.824746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13348.900586] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13351.506583] Lustre: server umount lustre-MDT0000 complete [13354.960674] LustreError: 448064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789278307 with bad export cookie 978620700138554114 [13354.977585] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13354.977895] LustreError: 448064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13355.350895] Lustre: server umount lustre-MDT0001 complete [13356.263588] LNet: 1 local NIs in recovery (showing 1): 192.168.203.105@tcp [13365.269258] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 01:45:15 (1789278315) [13982.150324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13982.537320] LustreError: 454517:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13982.593411] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13986.789127] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [13987.826605] LustreError: 454518:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13991.907458] LustreError: 454518:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13994.473648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13994.741696] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13998.495938] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [14004.927730] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14010.218628] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14029.353424] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14029.701827] Lustre: Found index 0 for lustre-OST0000, updating log [14030.203288] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14031.233170] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [14034.145695] LNet: 1 local NIs in recovery (showing 1): 192.168.203.105@tcp [14037.524379] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [14046.527545] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14049.761887] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14049.795596] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14049.803801] Lustre: Skipped 4 previous similar messages [14055.018428] Lustre: server umount lustre-OST0000 complete [14059.558583] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14059.563391] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14059.576143] Lustre: Skipped 1 previous similar message [14059.590539] Lustre: Skipped 1 previous similar message [14060.641352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14060.649662] Lustre: Skipped 1 previous similar message [14062.690827] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14062.695020] Lustre: Skipped 3 previous similar messages [14064.753702] LustreError: 454512:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14064.787384] LustreError: 454512:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [14064.837561] Lustre: server umount lustre-MDT0000 complete [14068.641070] LustreError: 454500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789279020 with bad export cookie 978620700138555206 [14068.654383] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14068.657536] LustreError: 454500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [14069.061968] Lustre: server umount lustre-MDT0001 complete [14094.559561] LNet: 1 local NIs in recovery (showing 1): 192.168.203.105@tcp [14097.563635] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_hostid [14106.740956] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [14140.356802] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 01:58:10 (1789279090) [14156.831887] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [14159.389391] Key type lgssc unregistered [14159.707124] LNet: 461132:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14159.720937] LNetError: 461132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14284.644677] LNet: Removed LNI 192.168.203.105@tcp [14285.484534] Key type .llcrypt unregistered [14285.497050] Key type ._llcrypt unregistered [14301.844779] Key type ._llcrypt registered [14301.847156] Key type .llcrypt registered [14301.978544] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [14302.691928] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14302.715749] alg: No test for adler32 (adler32-zlib) [14303.828512] Lustre: Lustre: Build Version: 2.17.57_105_g2595dd0 [14304.093845] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [14305.792173] Key type lgssc registered [14306.767783] Lustre: Echo OBD driver; http://www.lustre.org/ [14311.627450] LNet: Added LNI 192.168.203.105@tcp100 [8/256/0/180] [14312.587570] LNet: Added LNI 192.168.203.105@tcp101 [8/256/0/180] [14313.428644] LNet: Added LNI 192.168.203.105@tcp102 [8/256/0/180] [14314.239710] LNet: Added LNI 192.168.203.105@tcp103 [8/256/0/180] [14314.968482] LNet: Added LNI 192.168.203.105@tcp104 [8/256/0/180] [14315.878791] LNet: Added LNI 192.168.203.105@tcp105 [8/256/0/180] [14316.808980] LNet: Added LNI 192.168.203.105@tcp106 [8/256/0/180] [14317.831566] LNet: Added LNI 192.168.203.105@tcp107 [8/256/0/180] [14318.746937] LNet: Added LNI 192.168.203.105@tcp108 [8/256/0/180] [14319.678777] LNet: Added LNI 192.168.203.105@tcp109 [8/256/0/180] [14320.560991] LNet: Added LNI 192.168.203.105@tcp110 [8/256/0/180] [14321.463781] LNet: Added LNI 192.168.203.105@tcp111 [8/256/0/180] [14322.406356] LNet: Added LNI 192.168.203.105@tcp112 [8/256/0/180] [14323.313957] LNet: Added LNI 192.168.203.105@tcp113 [8/256/0/180] [14324.421647] LNet: Added LNI 192.168.203.105@tcp114 [8/256/0/180] [14325.421286] LNet: Added LNI 192.168.203.105@tcp115 [8/256/0/180] [14326.417322] LNet: Added LNI 192.168.203.105@tcp116 [8/256/0/180] [14327.415885] LNet: Added LNI 192.168.203.105@tcp117 [8/256/0/180] [14328.248081] LNet: Added LNI 192.168.203.105@tcp118 [8/256/0/180] [14329.159520] LNet: Added LNI 192.168.203.105@tcp119 [8/256/0/180] [14329.999443] LNet: Added LNI 192.168.203.105@tcp120 [8/256/0/180] [14330.840371] LNet: Added LNI 192.168.203.105@tcp121 [8/256/0/180] [14331.598440] LNet: Added LNI 192.168.203.105@tcp122 [8/256/0/180] [14332.313142] LNet: Added LNI 192.168.203.105@tcp123 [8/256/0/180] [14333.277487] LNet: Added LNI 192.168.203.105@tcp124 [8/256/0/180] [14334.143952] LNet: Added LNI 192.168.203.105@tcp125 [8/256/0/180] [14334.882914] LNet: Added LNI 192.168.203.105@tcp126 [8/256/0/180] [14335.714731] LNet: Added LNI 192.168.203.105@tcp127 [8/256/0/180] [14336.731566] LNet: Added LNI 192.168.203.105@tcp128 [8/256/0/180] [14337.651642] LNet: Added LNI 192.168.203.105@tcp129 [8/256/0/180] [14338.753152] LNet: Added LNI 192.168.203.105@tcp130 [8/256/0/180] [14339.593592] LNet: Added LNI 192.168.203.105@tcp131 [8/256/0/180] [14340.457907] LNet: Added LNI 192.168.203.105@tcp132 [8/256/0/180] [14341.292649] LNet: Added LNI 192.168.203.105@tcp133 [8/256/0/180] [14342.432655] LNet: Added LNI 192.168.203.105@tcp134 [8/256/0/180] [14343.384462] LNet: Added LNI 192.168.203.105@tcp135 [8/256/0/180] [14350.448977] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14350.504413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14351.630662] Lustre: 464175:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.105@tcp: can't add NID 192.168.203.105@tcp130: rc = -75 [14351.718842] LNetError: 461623:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.105 [14351.750199] LNetError: Protocol error connecting to 192.168.203.105@tcp120 at host 192.168.203.105:988: is it running a compatible version of Lustre? [14351.769676] LNetError: Skipped 4 previous similar messages [14351.817890] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14351.861906] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14351.926951] Lustre: lustre-MDT0000: new disk, initializing [14352.009110] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14352.019982] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14352.223501] LNet: 1 local NIs in recovery (showing 1): 192.168.203.105@tcp122 [14352.225271] LNetError: 461621:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.105 [14352.242775] LNetError: 461621:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 7 previous similar messages [14352.252986] LNetError: Protocol error connecting to 192.168.203.105@tcp104 at host 192.168.203.105:988: is it running a compatible version of Lustre? [14352.264037] LNetError: Skipped 2 previous similar messages [14354.290354] LNetError: 461621:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.105 [14354.303817] LNetError: 461621:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 7 previous similar messages [14354.309570] LNetError: Protocol error connecting to 192.168.203.105@tcp122 at host 192.168.203.105:988: is it running a compatible version of Lustre? [14354.320183] LNetError: Skipped 7 previous similar messages [14356.835990] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [14358.370479] LNetError: 461623:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.105 [14358.387797] LNetError: Protocol error connecting to 192.168.203.105@tcp122 at host 192.168.203.105:988: is it running a compatible version of Lustre? [14366.567165] LNetError: 464201:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.105 [14366.579674] LNetError: Protocol error connecting to 192.168.203.105@tcp122 at host 192.168.203.105:988: is it running a compatible version of Lustre? [14368.029775] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14368.154636] Lustre: 464191:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14368.245936] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14368.252698] Lustre: Skipped 1 previous similar message [14368.350639] Lustre: lustre-MDT0001: new disk, initializing [14368.391601] Lustre: 464995:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.105@tcp: can't add NID 192.168.203.105@tcp130: rc = -75 [14368.407700] Lustre: 464995:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14368.500787] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14368.533497] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14368.548338] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14373.173617] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [14377.014618] Lustre: 465419:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.105@tcp: can't add NID 192.168.203.105@tcp130: rc = -75 [14377.029748] Lustre: 465419:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14381.921758] LNetError: 464196:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.105 [14381.928745] LNetError: Protocol error connecting to 192.168.203.105@tcp109 at host 192.168.203.105:988: is it running a compatible version of Lustre? [14383.257489] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14389.752887] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14396.155594] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14396.440919] Lustre: lustre-OST0000: new disk, initializing [14396.447093] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14396.460982] Lustre: 466167:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14396.575489] Lustre: 466172:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.105@tcp: can't add NID 192.168.203.105@tcp130: rc = -75 [14396.586062] Lustre: 466172:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14396.666876] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14398.307043] LNetError: 464205:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.105 [14398.321851] LNetError: 464205:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 20 previous similar messages [14398.330958] LNetError: Protocol error connecting to 192.168.203.105@tcp128 at host 192.168.203.105:988: is it running a compatible version of Lustre? [14398.345093] LNetError: Skipped 20 previous similar messages [14401.580483] Lustre: 466418:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.105@tcp: can't add NID 192.168.203.105@tcp130: rc = -75 [14401.591943] Lustre: 466418:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14401.658420] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14401.668592] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14401.796203] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14403.483800] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [14412.639334] LNet: 1 local NIs in recovery (showing 1): 192.168.203.105@tcp122 [14413.907343] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14422.505447] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14422.509786] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14422.512955] LustreError: Skipped 1 previous similar message [14422.515338] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14422.535836] Lustre: Skipped 1 previous similar message [14427.645790] Lustre: server umount lustre-OST0000 complete [14432.099038] LNetError: 464208:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.105 [14432.108470] LNetError: 464208:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 26 previous similar messages [14432.124220] LNetError: Protocol error connecting to 192.168.203.105@tcp109 at host 192.168.203.105:988: is it running a compatible version of Lustre? [14432.145792] LNetError: Skipped 26 previous similar messages [14432.739310] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14432.742559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14432.755539] Lustre: Skipped 1 previous similar message [14432.771979] Lustre: Skipped 2 previous similar messages [14437.646151] Lustre: server umount lustre-MDT0000 complete [14440.918949] LustreError: 464184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789279393 with bad export cookie 10849378761305142349 [14440.933462] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14440.933738] LustreError: 464184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14441.271886] Lustre: server umount lustre-MDT0001 complete [14456.380331] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [14460.078681] Key type lgssc unregistered [14460.734821] LNet: 468082:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14460.741916] LNetError: 468082:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14460.750503] LNet: Removed LNI 192.168.203.105@tcp [14461.791204] LNet: Removed LNI 192.168.203.105@tcp100 [14461.798709] LNetError: 468082:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14461.805948] LNetError: 468082:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14462.879165] LNet: Removed LNI 192.168.203.105@tcp101 [14462.882262] LNetError: 468082:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14462.886604] LNet: Removed LNI 192.168.203.105@tcp102 [14463.903280] LNet: Removed LNI 192.168.203.105@tcp103 [14464.927201] LNet: Removed LNI 192.168.203.105@tcp104 [14464.930047] LNetError: 468082:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14464.938119] LNetError: 468082:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14464.954556] LNet: Removed LNI 192.168.203.105@tcp105 [14466.015161] LNet: Removed LNI 192.168.203.105@tcp106 [14467.039163] LNet: Removed LNI 192.168.203.105@tcp107 [14468.063290] LNet: Removed LNI 192.168.203.105@tcp108 [14469.087244] LNet: Removed LNI 192.168.203.105@tcp109 [14469.094575] LNetError: 468082:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14469.105454] LNetError: 468082:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14469.119441] LNet: Removed LNI 192.168.203.105@tcp110 [14470.175183] LNet: Removed LNI 192.168.203.105@tcp111 [14471.199303] LNet: Removed LNI 192.168.203.105@tcp112 [14472.289175] LNet: Removed LNI 192.168.203.105@tcp113 [14473.314400] LNet: Removed LNI 192.168.203.105@tcp114 [14474.335228] LNet: Removed LNI 192.168.203.105@tcp115 [14475.359718] LNet: Removed LNI 192.168.203.105@tcp116 [14476.383618] LNet: Removed LNI 192.168.203.105@tcp117 [14477.407347] LNet: Removed LNI 192.168.203.105@tcp118 [14477.409689] LNetError: 468082:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14477.432360] LNetError: 468082:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [14477.450524] LNet: Removed LNI 192.168.203.105@tcp119 [14478.495139] LNet: Removed LNI 192.168.203.105@tcp120 [14479.519148] LNet: Removed LNI 192.168.203.105@tcp121 [14479.521359] LNet: Removed LNI 192.168.203.105@tcp122 [14480.543152] LNet: Removed LNI 192.168.203.105@tcp123 [14481.567398] LNet: Removed LNI 192.168.203.105@tcp124 [14482.591181] LNet: Removed LNI 192.168.203.105@tcp125 [14483.615149] LNet: Removed LNI 192.168.203.105@tcp126 [14484.639213] LNet: Removed LNI 192.168.203.105@tcp127 [14485.663147] LNet: Removed LNI 192.168.203.105@tcp128 [14486.687245] LNet: Removed LNI 192.168.203.105@tcp129 [14486.692875] LNet: Removed LNI 192.168.203.105@tcp130 [14486.704727] LNet: Removed LNI 192.168.203.105@tcp131 [14486.711574] LNet: Removed LNI 192.168.203.105@tcp132 [14486.717802] LNet: Removed LNI 192.168.203.105@tcp133 [14486.725683] LNet: Removed LNI 192.168.203.105@tcp134 [14486.744743] LNet: Removed LNI 192.168.203.105@tcp135 [14487.605643] Key type .llcrypt unregistered [14487.611794] Key type ._llcrypt unregistered [14515.246854] Key type ._llcrypt registered [14515.249651] Key type .llcrypt registered [14515.393857] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_hostid [14534.024967] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [14535.770426] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14535.875047] alg: No test for adler32 (adler32-zlib) [14537.031215] Lustre: Lustre: Build Version: 2.17.57_105_g2595dd0 [14537.329121] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [14539.079131] Key type lgssc registered [14540.490936] Lustre: Echo OBD driver; http://www.lustre.org/ [14578.339690] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 02:05:28 (1789279528) [14598.747585] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [14602.380635] Key type lgssc unregistered [14602.710693] LNet: 472168:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14602.729947] LNetError: 472168:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14602.743838] LNet: Removed LNI 192.168.203.105@tcp [14603.865611] Key type .llcrypt unregistered [14603.869639] Key type ._llcrypt unregistered [14624.126419] Key type ._llcrypt registered [14624.129404] Key type .llcrypt registered [14624.216563] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [14625.291572] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14625.490052] alg: No test for adler32 (adler32-zlib) [14626.682313] Lustre: Lustre: Build Version: 2.17.57_105_g2595dd0 [14627.057024] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [14628.703204] Key type lgssc registered [14630.202704] Lustre: Echo OBD driver; http://www.lustre.org/ [14640.697540] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14640.733372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14641.919146] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14641.949806] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14642.024260] Lustre: lustre-MDT0000: new disk, initializing [14642.111337] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14642.125439] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14645.953983] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [14656.560208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14656.691892] Lustre: 473376:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14656.738914] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14656.743172] Lustre: Skipped 1 previous similar message [14656.796462] Lustre: lustre-MDT0001: new disk, initializing [14656.898855] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14656.945838] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14656.959479] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14662.379619] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [14671.918842] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14678.129539] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14684.633788] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14684.915190] Lustre: lustre-OST0000: new disk, initializing [14684.920199] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14684.927344] Lustre: 475346:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14685.011266] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14691.177427] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [14694.474398] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14694.485480] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14694.609412] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14700.670518] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14705.833108] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing lnet_if_list [14711.920264] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14711.955399] LNet: Added LNI 192.168.203.105@tcp10 [8/256/0/180] [14717.928279] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14717.973772] LNet: Added LNI 192.168.203.105@tcp11 [8/256/0/180] [14721.933712] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14721.964327] LNet: Added LNI 192.168.203.105@tcp12 [8/256/0/180] [14726.321760] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14726.348887] LNet: Added LNI 192.168.203.105@tcp13 [8/256/0/180] [14730.942367] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14730.986399] LNet: Added LNI 192.168.203.105@tcp14 [8/256/0/180] [14735.606562] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14735.643701] LNet: Added LNI 192.168.203.105@tcp15 [8/256/0/180] [14740.308931] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14740.338768] LNet: Added LNI 192.168.203.105@tcp16 [8/256/0/180] [14745.556790] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14745.597991] LNet: Added LNI 192.168.203.105@tcp17 [8/256/0/180] [14750.984070] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14751.070033] LNet: Added LNI 192.168.203.105@tcp18 [8/256/0/180] [14755.888218] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14755.918110] LNet: Added LNI 192.168.203.105@tcp19 [8/256/0/180] [14761.203349] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14761.234644] LNet: Added LNI 192.168.203.105@tcp20 [8/256/0/180] [14767.125199] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14767.183051] LNet: Added LNI 192.168.203.105@tcp21 [8/256/0/180] [14772.551253] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14772.576738] LNet: Added LNI 192.168.203.105@tcp22 [8/256/0/180] [14778.492816] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14778.531910] LNet: Added LNI 192.168.203.105@tcp23 [8/256/0/180] [14785.693253] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14785.714465] LNet: Added LNI 192.168.203.105@tcp24 [8/256/0/180] [14791.572378] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14791.618495] LNet: Added LNI 192.168.203.105@tcp25 [8/256/0/180] [14797.520516] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14797.554488] LNet: Added LNI 192.168.203.105@tcp26 [8/256/0/180] [14803.724834] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14803.761177] LNet: Added LNI 192.168.203.105@tcp27 [8/256/0/180] [14810.107812] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14810.139028] LNet: Added LNI 192.168.203.105@tcp28 [8/256/0/180] [14817.050973] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14817.084906] LNet: Added LNI 192.168.203.105@tcp29 [8/256/0/180] [14823.930253] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14823.988859] LNet: Added LNI 192.168.203.105@tcp30 [8/256/0/180] [14830.218648] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14830.250589] LNet: Added LNI 192.168.203.105@tcp31 [8/256/0/180] [14836.418609] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14836.443691] LNet: Added LNI 192.168.203.105@tcp32 [8/256/0/180] [14842.509176] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14842.553902] LNet: Added LNI 192.168.203.105@tcp33 [8/256/0/180] [14849.388852] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14849.415951] LNet: Added LNI 192.168.203.105@tcp34 [8/256/0/180] [14856.375993] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14856.425721] LNet: Added LNI 192.168.203.105@tcp35 [8/256/0/180] [14864.207831] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14864.237691] LNet: Added LNI 192.168.203.105@tcp36 [8/256/0/180] [14871.756566] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14871.804045] LNet: Added LNI 192.168.203.105@tcp37 [8/256/0/180] [14879.005421] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14879.034893] LNet: Added LNI 192.168.203.105@tcp38 [8/256/0/180] [14886.261448] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14886.285261] LNet: Added LNI 192.168.203.105@tcp39 [8/256/0/180] [14892.852315] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14892.891908] LNet: Added LNI 192.168.203.105@tcp40 [8/256/0/180] [14900.032616] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14900.055587] LNet: Added LNI 192.168.203.105@tcp41 [8/256/0/180] [14906.486953] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [14906.516641] LNet: Added LNI 192.168.203.105@tcp42 [8/256/0/180] [14913.293507] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [14913.313994] LNet: Added LNI 192.168.203.105@tcp43 [8/256/0/180] [14919.879895] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [14919.924747] LNet: Added LNI 192.168.203.105@tcp44 [8/256/0/180] [14928.261708] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [14928.289970] LNet: Added LNI 192.168.203.105@tcp45 [8/256/0/180] [14936.718079] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [14936.759895] LNet: Added LNI 192.168.203.105@tcp46 [8/256/0/180] [14946.462655] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [14946.481460] LNet: Added LNI 192.168.203.105@tcp47 [8/256/0/180] [14955.702646] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [14955.740584] LNet: Added LNI 192.168.203.105@tcp48 [8/256/0/180] [14964.410937] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [14964.450485] LNet: Added LNI 192.168.203.105@tcp49 [8/256/0/180] [14973.092774] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [14973.129811] LNet: Added LNI 192.168.203.105@tcp50 [8/256/0/180] [14994.923846] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14994.926961] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14994.934606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14994.950569] Lustre: Skipped 2 previous similar messages [14994.962107] Lustre: Skipped 2 previous similar messages [14998.797381] Lustre: server umount lustre-MDT0000 complete [15002.446469] LustreError: 473370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789279954 with bad export cookie 7208362892045358536 [15002.461819] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15002.777749] Lustre: server umount lustre-MDT0001 complete [15019.120768] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [15021.344446] Lustre: 472647:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789279957/real 1789279957] req@00000000d90f877d x1876195624000512/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789279973 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15021.380693] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15023.583275] Lustre: 472650:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789279959/real 1789279959] req@00000000cb42d32e x1876195624000640/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789279975 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15026.655306] Lustre: 472650:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789279962/real 1789279962] req@000000005cf7ba2f x1876195624000768/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789279978 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15026.951534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15027.755799] LustreError: 475338:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15027.772037] LustreError: 475338:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [15027.825264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15028.703192] Lustre: 472648:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789279964/real 1789279964] req@000000006a1b3f69 x1876195624001024/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789279980 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15030.815949] LustreError: 475339:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15031.847327] LustreError: 499581:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15031.875528] LustreError: 499581:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15031.925164] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [15033.829944] Lustre: 472648:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789279969/real 1789279969] req@00000000cb8a5873 x1876195624001152/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789279985 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15036.897268] LustreError: 499582:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15036.914728] LustreError: 499582:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15040.175734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15040.590737] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15044.937425] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [15045.619561] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15050.790379] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15052.391926] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15059.170111] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15081.448339] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15081.453868] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15081.466375] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15081.498402] Lustre: Skipped 1 previous similar message [15086.433981] Lustre: server umount lustre-OST0000 complete [15091.683021] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15091.687550] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15091.718344] Lustre: Skipped 1 previous similar message [15091.727972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15091.740105] Lustre: Skipped 1 previous similar message [15096.364730] Lustre: server umount lustre-MDT0000 complete [15096.803151] LustreError: 499582:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15096.845320] LustreError: 499582:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15100.440308] LustreError: 473370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789280052 with bad export cookie 7208362892045361105 [15100.442063] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15100.461954] LustreError: 473370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [15101.016935] Lustre: server umount lustre-MDT0001 complete [15111.222233] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [15113.741726] Key type lgssc unregistered [15114.162405] LNet: 502382:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15114.172575] LNetError: 502382:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15114.183124] LNet: Removed LNI 192.168.203.105@tcp [15114.186190] LNet: Removed LNI 192.168.203.105@tcp10 [15114.189375] LNet: Removed LNI 192.168.203.105@tcp11 [15114.192634] LNet: Removed LNI 192.168.203.105@tcp12 [15114.195457] LNet: Removed LNI 192.168.203.105@tcp13 [15114.201642] LNet: Removed LNI 192.168.203.105@tcp14 [15114.204607] LNet: Removed LNI 192.168.203.105@tcp15 [15114.207366] LNet: Removed LNI 192.168.203.105@tcp16 [15114.213902] LNet: Removed LNI 192.168.203.105@tcp17 [15114.221765] LNet: Removed LNI 192.168.203.105@tcp18 [15114.224604] LNet: Removed LNI 192.168.203.105@tcp19 [15114.230664] LNet: Removed LNI 192.168.203.105@tcp20 [15114.238128] LNet: Removed LNI 192.168.203.105@tcp21 [15114.247460] LNet: Removed LNI 192.168.203.105@tcp22 [15114.256695] LNet: Removed LNI 192.168.203.105@tcp23 [15114.260479] LNet: Removed LNI 192.168.203.105@tcp24 [15114.263641] LNet: Removed LNI 192.168.203.105@tcp25 [15114.268131] LNet: Removed LNI 192.168.203.105@tcp26 [15114.272531] LNet: Removed LNI 192.168.203.105@tcp27 [15114.276280] LNet: Removed LNI 192.168.203.105@tcp28 [15114.279635] LNet: Removed LNI 192.168.203.105@tcp29 [15114.282860] LNet: Removed LNI 192.168.203.105@tcp30 [15114.285199] LNet: Removed LNI 192.168.203.105@tcp31 [15114.288315] LNet: Removed LNI 192.168.203.105@tcp32 [15114.291432] LNet: Removed LNI 192.168.203.105@tcp33 [15114.294730] LNet: Removed LNI 192.168.203.105@tcp34 [15114.298335] LNet: Removed LNI 192.168.203.105@tcp35 [15114.301394] LNet: Removed LNI 192.168.203.105@tcp36 [15114.303615] LNet: Removed LNI 192.168.203.105@tcp37 [15114.306739] LNet: Removed LNI 192.168.203.105@tcp38 [15114.309274] LNet: Removed LNI 192.168.203.105@tcp39 [15114.312710] LNet: Removed LNI 192.168.203.105@tcp40 [15114.317610] LNet: Removed LNI 192.168.203.105@tcp41 [15114.319901] LNet: Removed LNI 192.168.203.105@tcp42 [15114.323334] LNet: Removed LNI 192.168.203.105@tcp43 [15114.326299] LNet: Removed LNI 192.168.203.105@tcp44 [15114.329259] LNet: Removed LNI 192.168.203.105@tcp45 [15114.332453] LNet: Removed LNI 192.168.203.105@tcp46 [15114.335594] LNet: Removed LNI 192.168.203.105@tcp47 [15114.338725] LNet: Removed LNI 192.168.203.105@tcp48 [15114.342131] LNet: Removed LNI 192.168.203.105@tcp49 [15114.355240] LNet: Removed LNI 192.168.203.105@tcp50 [15114.993307] Key type .llcrypt unregistered [15114.999733] Key type ._llcrypt unregistered [15137.911879] Key type ._llcrypt registered [15137.914716] Key type .llcrypt registered [15138.002574] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_hostid [15153.881656] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [15154.523476] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15154.620461] alg: No test for adler32 (adler32-zlib) [15155.599946] Lustre: Lustre: Build Version: 2.17.57_105_g2595dd0 [15155.787688] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [15157.423535] Key type lgssc registered [15158.709537] Lustre: Echo OBD driver; http://www.lustre.org/ [15198.528489] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15198.549261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15199.847923] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15199.877611] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15199.990766] Lustre: lustre-MDT0000: new disk, initializing [15200.092805] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15200.130811] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15205.030430] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [15214.612770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15214.742045] Lustre: 505577:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15214.825595] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15214.832279] Lustre: Skipped 1 previous similar message [15214.909348] Lustre: lustre-MDT0001: new disk, initializing [15214.999585] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15215.061702] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15215.079338] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15219.326867] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [15228.349638] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15233.875891] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15240.907666] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15241.057709] Lustre: *** cfs_fail_loc=90f, val=0*** [15241.108621] Lustre: lustre-OST0000: new disk, initializing [15241.113077] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15241.117906] Lustre: 507593:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15241.214143] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15245.461933] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [15247.413121] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15247.421493] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15247.505917] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15254.351471] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15263.200650] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15263.206091] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15263.214264] LustreError: Skipped 1 previous similar message [15263.215682] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15263.254912] Lustre: Skipped 1 previous similar message [15266.974256] Lustre: server umount lustre-OST0000 complete [15271.398345] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15271.406476] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15271.416518] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15271.419763] Lustre: Skipped 1 previous similar message [15273.444050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15273.454183] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15273.467271] Lustre: Skipped 1 previous similar message [15276.386310] Lustre: server umount lustre-MDT0000 complete [15281.106204] LustreError: 505570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789280233 with bad export cookie 11085708571722051304 [15281.110305] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15281.118784] LustreError: 505570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [15281.789133] Lustre: server umount lustre-MDT0001 complete [15296.847216] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [15300.802867] Key type lgssc unregistered [15301.132416] LNet: 509497:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15301.149487] LNetError: 509497:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15301.176791] LNet: Removed LNI 192.168.203.105@tcp [15302.314462] Key type .llcrypt unregistered [15302.322373] Key type ._llcrypt unregistered [15323.334237] Key type ._llcrypt registered [15323.336455] Key type .llcrypt registered [15323.430278] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [15324.255587] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15324.352168] alg: No test for adler32 (adler32-zlib) [15325.435442] Lustre: Lustre: Build Version: 2.17.57_105_g2595dd0 [15325.638860] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [15327.295176] Key type lgssc registered [15328.351854] Lustre: Echo OBD driver; http://www.lustre.org/ [15335.524274] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15335.547205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15337.133853] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15342.076048] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [15347.690755] LustreError: 510763:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15347.723059] LustreError: 510763:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [15351.864979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15352.340215] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15357.008556] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [15363.247793] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15368.068353] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15374.807796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15375.246832] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15381.541899] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [15388.696501] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15399.016750] loop0: detected capacity change from 0 to 204800000 [15399.162814] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15399.177674] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15399.199989] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15404.394320] loop0: detected capacity change from 0 to 204800000 [15404.523776] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15404.537536] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15404.553967] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15408.911377] loop0: detected capacity change from 0 to 204800000 [15413.786514] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15413.893950] Lustre: fs15246-OST0000: new disk, initializing [15413.905573] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15413.919694] Lustre: 514358:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15413.992314] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15420.123145] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [15430.696890] loop1: detected capacity change from 0 to 204800000 [15436.371643] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15436.466869] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15436.522747] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15436.573891] Lustre: fs15246-MDT0000: new disk, initializing [15436.728821] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15436.756316] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15436.870716] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15437.921039] Lustre: 515480:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000004b34ea77 x1876196355779840/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1789280406 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15440.843895] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [15442.989673] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15443.000457] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15443.043150] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15462.902585] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15462.928636] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15465.638521] Lustre: server umount fs15246-MDT0000 complete [15468.006862] LustreError: 510760:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15473.135356] LustreError: 515806:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15474.813345] Lustre: server umount fs15246-OST0000 complete [15488.490966] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15488.510967] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15488.525378] Lustre: Skipped 1 previous similar message [15492.292888] Lustre: server umount lustre-OST0000 complete [15498.722425] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15498.724769] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15498.735534] Lustre: Skipped 2 previous similar messages [15498.756832] Lustre: Skipped 1 previous similar message [15502.185596] Lustre: server umount lustre-MDT0000 complete [15503.843302] LustreError: 511492:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15503.871240] LustreError: 511492:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15506.049863] LustreError: 510744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789280458 with bad export cookie 18089783190331958479 [15506.059692] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15506.072481] LustreError: 510744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [15506.528354] Lustre: server umount lustre-MDT0001 complete [15517.856521] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [15520.753852] Key type lgssc unregistered [15521.062827] LNet: 518577:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15521.072234] LNetError: 518577:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15522.090315] LNet: Removed LNI 192.168.203.105@tcp [15522.916698] Key type .llcrypt unregistered [15522.919475] Key type ._llcrypt unregistered [15540.963412] Key type ._llcrypt registered [15540.965761] Key type .llcrypt registered [15541.047364] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [15541.595695] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15541.620102] alg: No test for adler32 (adler32-zlib) [15542.686280] Lustre: Lustre: Build Version: 2.17.57_105_g2595dd0 [15542.902789] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [15544.551422] Key type lgssc registered [15545.569702] Lustre: Echo OBD driver; http://www.lustre.org/ [15581.761761] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_hostid [15594.032515] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [15632.462407] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15632.502214] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15633.881443] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15633.916175] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15634.013862] Lustre: lustre-MDT0000: new disk, initializing [15634.092866] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15634.105126] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15638.595477] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [15649.495682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15649.599112] Lustre: 523058:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15649.652273] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15649.656061] Lustre: Skipped 1 previous similar message [15649.705477] Lustre: lustre-MDT0001: new disk, initializing [15649.809162] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15649.850596] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15649.866823] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15653.815695] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [15663.522315] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15670.673940] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15679.970486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15680.330856] Lustre: lustre-OST0000: new disk, initializing [15680.337672] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15680.348204] Lustre: 525026:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15680.432128] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15686.691775] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15686.702124] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15686.824779] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15687.189039] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [15697.202270] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15701.422635] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15701.657318] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15705.837839] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15706.022088] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15712.225686] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15712.231494] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15712.251787] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15712.737980] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15712.741662] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15712.781281] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15714.152395] Lustre: server umount lustre-OST0000 complete [15717.860549] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15717.883675] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15717.892356] Lustre: Skipped 1 previous similar message [15722.991795] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15723.002468] Lustre: Skipped 1 previous similar message [15728.097794] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15728.104758] Lustre: Skipped 1 previous similar message [15732.191162] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15732.383188] Lustre: server umount lustre-MDT0000 complete [15735.881119] LustreError: 523048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789280688 with bad export cookie 9768764571879091050 [15735.893633] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15735.898799] LustreError: 523048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15736.287869] Lustre: server umount lustre-MDT0001 complete [15760.382447] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_hostid [15771.638870] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [15803.383863] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15498 sec ========= 02:25:53 (1789280753) [15804.955800] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 02:25:55 (1789280755) === [15808.772214] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 02:25:59 (1789280759) === [15851.199586] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [15854.336311] Key type lgssc unregistered [15854.629719] LNet: 531670:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15854.635285] LNetError: 531670:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15855.719989] LNet: Removed LNI 192.168.203.105@tcp [15856.998815] Key type .llcrypt unregistered [15857.007655] Key type ._llcrypt unregistered