[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 462085418 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2891144K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001015] APIC: Switch to symmetric I/O mode setup [ 0.002393] x2apic enabled [ 0.003012] Switched APIC routing to physical x2apic. [ 0.004016] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008013] pid_max: default: 32768 minimum: 301 [ 0.009140] LSM: Security Framework initializing [ 0.010056] Yama: becoming mindful. [ 0.011046] SELinux: Initializing. [ 0.012081] *** VALIDATE selinux *** [ 0.020663] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024567] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025159] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027046] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028121] *** VALIDATE tmpfs *** [ 0.030496] *** VALIDATE proc *** [ 0.032033] *** VALIDATE cgroup *** [ 0.033013] *** VALIDATE cgroup2 *** [ 0.035270] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036149] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038029] Spectre V2 : User space: Vulnerable [ 0.039009] Speculative Store Bypass: Vulnerable [ 0.042088] debug: unmapping init [mem 0xffffffffbd459000-0xffffffffbd460fff] [ 0.044974] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045716] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046026] ... version: 2 [ 0.047012] ... bit width: 48 [ 0.048012] ... generic registers: 4 [ 0.049012] ... value mask: 0000ffffffffffff [ 0.050016] ... max period: 00007fffffffffff [ 0.051017] ... fixed-purpose events: 3 [ 0.052014] ... event mask: 000000070000000f [ 0.053295] rcu: Hierarchical SRCU implementation. [ 0.055379] smp: Bringing up secondary CPUs ... [ 0.056567] x86: Booting SMP configuration: [ 0.057022] .... node #0, CPUs: #1 #2 #3 [ 0.060547] smp: Brought up 1 node, 4 CPUs [ 0.062015] smpboot: Max logical packages: 1 [ 0.063022] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.235944] node 0 deferred pages initialised in 171ms [ 0.238202] devtmpfs: initialized [ 0.239233] x86/mm: Memory block size: 128MB [ 0.241763] gcov: version magic: 0x41383552 [ 0.243213] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.244086] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.245274] pinctrl core: initialized pinctrl subsystem [ 0.246223] [ 0.246813] ************************************************************* [ 0.247016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.248014] ** ** [ 0.249016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.250011] ** ** [ 0.251011] ** This means that this kernel is built to expose internal ** [ 0.252015] ** IOMMU data structures, which may compromise security on ** [ 0.253016] ** your system. ** [ 0.254016] ** ** [ 0.255016] ** If you see this message and you are not debugging the ** [ 0.256016] ** kernel, report this immediately to your vendor! ** [ 0.257016] ** ** [ 0.258018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.259018] ************************************************************* [ 0.260790] NET: Registered protocol family 16 [ 0.261446] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.262059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.263065] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.264445] cpuidle: using governor menu [ 0.265000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.267569] PCI: Using configuration type 1 for base access [ 0.269127] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.278207] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.281125] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.285089] cryptd: max_cpu_qlen set to 1000 [ 0.288274] ACPI: Added _OSI(Module Device) [ 0.290020] ACPI: Added _OSI(Processor Device) [ 0.292018] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.294017] ACPI: Added _OSI(Processor Aggregator Device) [ 0.299110] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.306551] ACPI: Interpreter enabled [ 0.307076] ACPI: PM: (supports S0 S3 S4 S5) [ 0.308012] ACPI: Using IOAPIC for interrupt routing [ 0.310164] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.312413] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.323000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.325057] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.327024] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.329110] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.334567] acpiphp: Slot [2] registered [ 0.335153] acpiphp: Slot [5] registered [ 0.337126] acpiphp: Slot [6] registered [ 0.338127] acpiphp: Slot [7] registered [ 0.339158] acpiphp: Slot [8] registered [ 0.340174] acpiphp: Slot [9] registered [ 0.341131] acpiphp: Slot [10] registered [ 0.343134] acpiphp: Slot [3] registered [ 0.344099] acpiphp: Slot [4] registered [ 0.345091] acpiphp: Slot [11] registered [ 0.346149] acpiphp: Slot [12] registered [ 0.347122] acpiphp: Slot [13] registered [ 0.348102] acpiphp: Slot [14] registered [ 0.350101] acpiphp: Slot [15] registered [ 0.351092] acpiphp: Slot [16] registered [ 0.352096] acpiphp: Slot [17] registered [ 0.353156] acpiphp: Slot [18] registered [ 0.354113] acpiphp: Slot [19] registered [ 0.355125] acpiphp: Slot [20] registered [ 0.357103] acpiphp: Slot [21] registered [ 0.358091] acpiphp: Slot [22] registered [ 0.359096] acpiphp: Slot [23] registered [ 0.360155] acpiphp: Slot [24] registered [ 0.361127] acpiphp: Slot [25] registered [ 0.362107] acpiphp: Slot [26] registered [ 0.363124] acpiphp: Slot [27] registered [ 0.364109] acpiphp: Slot [28] registered [ 0.366109] acpiphp: Slot [29] registered [ 0.367094] acpiphp: Slot [30] registered [ 0.368139] acpiphp: Slot [31] registered [ 0.369066] PCI host bridge to bus 0000:00 [ 0.370021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.372024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.374026] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.376076] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.378029] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.380030] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.381197] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.384184] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.387440] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.397021] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.401954] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.405026] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.407024] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.410028] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.414140] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.416840] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.420058] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.423861] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.427840] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.437023] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.442022] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.449626] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.457018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.462018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.478016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.487524] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.496020] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.503025] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.520026] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.530908] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.536016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.542019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.567019] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.580740] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.589021] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.596020] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.617026] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.630043] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.637016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.643016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.658020] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.667850] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.677022] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.684022] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.702023] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.717029] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.720495] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.723423] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.726375] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.728233] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.734053] iommu: Default domain type: Passthrough [ 0.735659] SCSI subsystem initialized [ 0.737191] ACPI: bus type USB registered [ 0.739149] usbcore: registered new interface driver usbfs [ 0.741142] usbcore: registered new interface driver hub [ 0.743130] usbcore: registered new device driver usb [ 0.745247] pps_core: LinuxPPS API ver. 1 registered [ 0.748020] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.752096] PTP clock support registered [ 0.754118] EDAC MC: Ver: 3.0.0 [ 0.756156] PCI: Using ACPI for IRQ routing [ 0.759009] NetLabel: Initializing [ 0.760013] NetLabel: domain hash size = 128 [ 0.761014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.763115] NetLabel: unlabeled traffic allowed by default [ 0.766170] vgaarb: loaded [ 0.767323] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.769019] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.773459] clocksource: Switched to clocksource kvm-clock [ 0.883491] VFS: Disk quotas dquot_6.6.0 [ 0.885199] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.887904] *** VALIDATE ramfs *** [ 0.889408] *** VALIDATE hugetlbfs *** [ 0.891880] pnp: PnP ACPI init [ 0.894454] pnp: PnP ACPI: found 6 devices [ 0.911540] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.915065] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.917454] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.919863] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.922526] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.925130] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.928158] NET: Registered protocol family 2 [ 0.930582] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.935627] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.939591] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.945378] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.948753] TCP: Hash tables configured (established 65536 bind 65536) [ 0.951694] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.955942] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.958519] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.961606] NET: Registered protocol family 1 [ 0.964374] RPC: Registered named UNIX socket transport module. [ 0.966650] RPC: Registered udp transport module. [ 0.968675] RPC: Registered tcp transport module. [ 0.970800] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.973217] NET: Registered protocol family 44 [ 0.975050] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.977356] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.979209] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.981343] PCI: CLS 0 bytes, default 64 [ 0.983300] Unpacking initramfs... [ 2.429789] debug: unmapping init [mem 0xffff8e0ebcc54000-0xffff8e0ebffbffff] [ 2.434278] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.436253] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.439712] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.939757] Initialise system trusted keyrings [ 2.941390] Key type blacklist registered [ 2.943370] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.952195] zbud: loaded [ 2.955257] *** VALIDATE nfs *** [ 2.956735] *** VALIDATE nfs4 *** [ 2.958247] pstore: using deflate compression [ 2.961649] Platform Keyring initialized [ 3.065193] NET: Registered protocol family 38 [ 3.066776] Key type asymmetric registered [ 3.068239] Asymmetric key parser 'x509' registered [ 3.069747] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.072353] io scheduler mq-deadline registered [ 3.074327] io scheduler kyber registered [ 3.076244] io scheduler bfq registered [ 3.078355] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.082057] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.085507] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.088836] ACPI: Power Button [PWRF] [ 3.094388] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.100633] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.111326] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.116871] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.128914] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.161494] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.198293] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.203573] Non-volatile memory driver v1.3 [ 3.205362] Linux agpgart interface v0.103 [ 3.238307] virtio_blk virtio1: [vda] 146640 512-byte logical blocks (75.1 MB/71.6 MiB) [ 3.241545] vda: detected capacity change from 0 to 75079680 [ 3.258300] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.261103] vdb: detected capacity change from 0 to 1073741824 [ 3.277830] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.281243] vdc: detected capacity change from 0 to 2621440000 [ 3.297667] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.299980] vdd: detected capacity change from 0 to 2621440000 [ 3.315643] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.318918] vde: detected capacity change from 0 to 4294967296 [ 3.333695] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.336365] vdf: detected capacity change from 0 to 4294967296 [ 3.351344] libphy: Fixed MDIO Bus: probed [ 3.358873] usbcore: registered new interface driver usbserial_generic [ 3.361155] usbserial: USB Serial support registered for generic [ 3.363247] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.367187] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.368948] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.371533] mousedev: PS/2 mouse device common for all mice [ 3.374485] rtc_cmos 00:05: RTC can wake from S4 [ 3.377700] rtc_cmos 00:05: registered as rtc0 [ 3.379337] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.382235] intel_pstate: CPU model not supported [ 3.385352] hid: raw HID events driver (C) Jiri Kosina [ 3.387779] usbcore: registered new interface driver usbhid [ 3.387817] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.389108] usbhid: USB HID core driver [ 3.393725] drop_monitor: Initializing network drop monitor service [ 3.395163] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.396272] Initializing XFRM netlink socket [ 3.400493] NET: Registered protocol family 10 [ 3.400601] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.403201] Segment Routing with IPv6 [ 3.404468] NET: Registered protocol family 17 [ 3.406546] mpls_gso: MPLS GSO support [ 3.412421] RAS: Correctable Errors collector initialized. [ 3.414472] AVX version of gcm_enc/dec engaged. [ 3.416081] AES CTR mode by8 optimization enabled [ 3.488898] sched_clock: Marking stable (3488873904, 0)->(4423779568, -934905664) [ 3.492445] registered taskstats version 1 [ 3.494375] Loading compiled-in X.509 certificates [ 3.496484] zswap: loaded using pool lzo/zbud [ 3.522771] Key type big_key registered [ 3.536539] Key type encrypted registered [ 3.537884] ima: No TPM chip found, activating TPM-bypass! [ 3.539439] ima: Allocated hash algorithm: sha1 [ 3.540801] ima: No architecture policies found [ 3.542172] evm: Initialising EVM extended attributes: [ 3.543663] evm: security.selinux [ 3.544565] evm: security.ima [ 3.545387] evm: security.capability [ 3.546577] evm: HMAC attrs: 0x1 [ 3.548798] rtc_cmos 00:05: setting system clock to 2026-09-02 06:10:53 UTC (1788329453) [ 3.554398] debug: unmapping init [mem 0xffffffffbe403000-0xffffffffbe5fffff] [ 3.557104] debug: unmapping init [mem 0xffffffffbd182000-0xffffffffbd458fff] [ 3.565088] Write protecting the kernel read-only data: 28672k [ 3.568035] debug: unmapping init [mem 0xffffffffbb803000-0xffffffffbb9fffff] [ 3.570506] debug: unmapping init [mem 0xffffffffbc114000-0xffffffffbc1fffff] [ 3.601930] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.608976] systemd[1]: Detected virtualization kvm. [ 3.610652] systemd[1]: Detected architecture x86-64. [ 3.612382] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.635470] systemd[1]: No hostname configured. [ 3.637029] systemd[1]: Set hostname to . [ 3.638747] random: systemd: uninitialized urandom read (16 bytes read) [ 3.640766] systemd[1]: Initializing machine ID from random generator. [ 3.778674] random: systemd: uninitialized urandom read (16 bytes read) [ 3.780710] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.785103] random: systemd: uninitialized urandom read (16 bytes read) [ 3.787992] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.791647] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.423502] device-mapper: uevent: version 1.0.3 [ 4.425670] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.215565] random: fast init done [ 5.251946] virtio_net virtio0 ens2: renamed from eth0 [ 5.369305] scsi host0: ata_piix [ 5.396289] scsi host1: ata_piix [ 5.397792] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.400231] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.062472] dracut-initqueue[581]: RTNETLINK answers: File exists [ 10.055297] random: crng init done [ 10.056710] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.534142] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.724784] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.976875] SELinux: Disabled at runtime. [ 12.032901] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.041151] systemd[1]: Detected virtualization kvm. [ 12.043140] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.598480] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.601922] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.612278] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.616706] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.620537] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.629193] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.633834] systemd[1]: Listening on initctl Compatibility Named Pipe. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Huge Pages File System... Mounting POSIX Message Queue File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-getty.slice. Mounting Kernel Debug File System... Starting Apply Kernel Variables... [ OK ] Stopped target Initrd File Systems. Activating swap /dev/disk/by-label/SWAP... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Journal Service.[ 12.830256] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 13.123306] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.516948] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.550803] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.730090] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.740234] EDAC sbridge: Ver: 1.1.2 [ 15.574920] Key type dns_resolver registered [ 15.886323] NFS: Registering the id_resolver key type [ 15.888603] Key type id_resolver registered [ 15.890563] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started dnf makecache --timer. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg360-server login: [ 95.377590] libcfs: loading out-of-tree module taints kernel. [ 95.443798] Key type ._llcrypt registered [ 95.446111] Key type .llcrypt registered [ 95.583878] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_hostid [ 114.676360] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 116.342379] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 116.353734] alg: No test for adler32 (adler32-zlib) [ 117.839301] Lustre: Lustre: Build Version: 2.17.57_104_g6cc0f2c [ 118.855270] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 120.767180] Key type lgssc registered [ 122.999212] Lustre: Echo OBD driver; http://www.lustre.org/ [ 141.238051] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 153.466525] hrtimer: interrupt took 3109180 ns [ 186.701653] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 200.568037] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 200.609229] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 202.010788] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 202.050912] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 202.162466] Lustre: lustre-MDT0000: new disk, initializing [ 202.293853] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 202.345464] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 207.039500] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 220.091262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 220.227121] Lustre: 6517:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 220.298991] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 220.305561] Lustre: Skipped 1 previous similar message [ 220.467236] Lustre: lustre-MDT0001: new disk, initializing [ 220.573861] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 220.617123] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 220.624992] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 225.216645] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 230.006780] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 239.481654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 239.864063] Lustre: lustre-OST0000: new disk, initializing [ 239.868256] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 239.892629] Lustre: 8454:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 240.003323] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 244.789552] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 244.800432] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 244.875846] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 246.219103] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 262.064243] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 262.260751] Lustre: lustre-OST0001: new disk, initializing [ 262.267312] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 262.275667] Lustre: 9527:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 262.375601] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 269.041602] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 270.350231] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 270.357700] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 270.401628] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 279.809887] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 284.873957] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 291.134726] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing check_logdir /tmp/testlogs/ [ 295.750678] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing yml_node [ 301.273672] Lustre: DEBUG MARKER: Client: 2.17.57.104 [ 304.235273] Lustre: DEBUG MARKER: MDS: 2.17.57.104 [ 306.919084] Lustre: DEBUG MARKER: OSS: 2.17.57.104 [ 308.382519] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Sep 2 02:15:57 EDT 2026 [ 326.123366] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 327.818486] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 347.106887] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 347.110871] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 347.126235] Lustre: Skipped 1 previous similar message [ 347.143208] Lustre: Skipped 2 previous similar messages [ 352.225555] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 352.230356] Lustre: Skipped 3 previous similar messages [ 357.350708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 357.367435] Lustre: Skipped 3 previous similar messages [ 358.368776] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 358.555411] Lustre: server umount lustre-MDT0000 complete [ 362.501614] LustreError: 6507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788329812 with bad export cookie 12271504422737264734 [ 362.510930] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 362.520060] LustreError: 6507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 367.588035] LustreError: 12109:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 367.589553] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 367.594152] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 367.601080] LustreError: 12109:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 367.625372] Lustre: Skipped 3 previous similar messages [ 368.944207] Lustre: server umount lustre-MDT0001 complete [ 372.867555] Lustre: server umount lustre-OST0000 complete [ 376.135994] Lustre: server umount lustre-OST0001 complete [ 383.227941] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_hostid [ 392.405738] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 427.720171] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 427.924111] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 427.955904] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 428.052692] Lustre: lustre-MDT0000: new disk, initializing [ 428.193737] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 428.224152] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 432.310936] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 441.658313] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 441.802940] Lustre: 15080:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 441.881097] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 441.891425] Lustre: Skipped 1 previous similar message [ 441.974300] Lustre: lustre-MDT0001: new disk, initializing [ 442.099551] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 442.152547] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 442.163775] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 446.199369] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 454.341446] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 458.928558] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 464.718775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 464.933808] Lustre: lustre-OST0000: new disk, initializing [ 464.937198] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 464.942301] Lustre: 17049:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 465.031486] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 466.576909] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 466.593432] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 466.710515] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 470.399792] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 478.629599] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 483.163763] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 483.426675] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 487.752702] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 488.024658] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 492.516174] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 492.522338] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 492.528653] LustreError: Skipped 1 previous similar message [ 492.552064] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 492.564270] Lustre: Skipped 2 previous similar messages [ 502.761252] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 502.770853] Lustre: Skipped 3 previous similar messages [ 504.287210] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 504.430587] Lustre: server umount lustre-OST0000 complete [ 508.909637] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 508.917067] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 508.930969] Lustre: Skipped 1 previous similar message [ 512.994317] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 514.456641] Lustre: server umount lustre-MDT0000 complete [ 517.352595] LustreError: 15074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788329967 with bad export cookie 12271504422737266267 [ 517.354193] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 517.365104] LustreError: 15074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 517.736480] Lustre: server umount lustre-MDT0001 complete [ 521.588542] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 02:19:31 (1788329971) [ 523.960531] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 525.626498] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 02:19:34 (1788329974) [ 531.343709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 531.693954] LustreError: 19458:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 531.767096] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 535.724879] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 537.059430] LustreError: 19459:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 541.156670] LustreError: 19458:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 543.410153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 547.981771] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 554.996895] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 560.268194] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 566.445995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 572.663454] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 580.409513] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 589.280178] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 589.283851] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 589.291136] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 589.293875] Lustre: Skipped 3 previous similar messages [ 592.360154] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 594.611642] Lustre: server umount lustre-OST0000 complete [ 600.033641] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 612.319952] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 612.550708] Lustre: server umount lustre-MDT0000 complete [ 612.833876] LustreError: 19454:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 612.855015] LustreError: 19454:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 616.752831] LustreError: 19439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788330066 with bad export cookie 12271504422737267149 [ 616.764903] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 617.235212] Lustre: server umount lustre-MDT0001 complete [ 630.309718] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 633.169498] Key type lgssc unregistered [ 633.420924] LNet: 22988:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 633.433902] LNetError: 22988:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 633.465215] LNet: Removed LNI 192.168.203.160@tcp [ 634.245127] Key type .llcrypt unregistered [ 634.249084] Key type ._llcrypt unregistered [ 655.934500] Key type ._llcrypt registered [ 655.936965] Key type .llcrypt registered [ 656.044831] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 656.817591] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 656.835446] alg: No test for adler32 (adler32-zlib) [ 657.966850] Lustre: Lustre: Build Version: 2.17.57_104_g6cc0f2c [ 658.267650] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 659.983285] Key type lgssc registered [ 661.063125] Lustre: Echo OBD driver; http://www.lustre.org/ [ 667.499909] loop: module loaded [ 667.588965] loop0: detected capacity change from 0 to 204800000 [ 667.773973] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 667.780474] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 667.938882] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 674.350407] loop0: detected capacity change from 0 to 204800000 [ 674.540793] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 674.553153] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 674.574472] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 679.784638] loop0: detected capacity change from 0 to 204800000 [ 685.295959] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 685.356792] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 686.652899] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 686.744767] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 686.926941] Lustre: lustre-MDT0000: new disk, initializing [ 687.171146] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 687.207857] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 692.661270] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 702.232188] loop1: detected capacity change from 0 to 204800000 [ 709.075035] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 709.682675] Lustre: lustre-OST07c6: new disk, initializing [ 709.687835] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 709.693701] Lustre: Skipped 1 previous similar message [ 709.698248] Lustre: 26174:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 709.810703] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 716.290688] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 718.456712] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 718.477328] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 718.551085] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 721.998370] Lustre: 25161:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 722.010138] Lustre: 25161:0:(osd_handler.c:2092:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 722.017184] Lustre: 25161:0:(osd_handler.c:2099:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 722.022697] Lustre: 25161:0:(osd_handler.c:2109:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 722.031587] Lustre: 25161:0:(osd_handler.c:2116:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 722.039884] Lustre: 25161:0:(osd_handler.c:2123:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 735.927937] Lustre: 25160:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 735.948605] Lustre: 25160:0:(osd_handler.c:2092:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 735.961228] Lustre: 25160:0:(osd_handler.c:2099:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 735.977599] Lustre: 25160:0:(osd_handler.c:2109:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 735.989945] Lustre: 25160:0:(osd_handler.c:2116:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 736.004625] Lustre: 25160:0:(osd_handler.c:2123:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 740.994336] Lustre: 26464:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 258 < left 278, rollback = 2 [ 741.017988] Lustre: 26464:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 1 previous similar message [ 741.027903] Lustre: 26464:0:(osd_handler.c:2092:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 741.035337] Lustre: 26464:0:(osd_handler.c:2092:osd_trans_dump_creds()) Skipped 1 previous similar message [ 741.043870] Lustre: 26464:0:(osd_handler.c:2099:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 741.053531] Lustre: 26464:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 1 previous similar message [ 741.060428] Lustre: 26464:0:(osd_handler.c:2109:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 741.070437] Lustre: 26464:0:(osd_handler.c:2109:osd_trans_dump_creds()) Skipped 1 previous similar message [ 741.076306] Lustre: 26464:0:(osd_handler.c:2116:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 741.085199] Lustre: 26464:0:(osd_handler.c:2116:osd_trans_dump_creds()) Skipped 1 previous similar message [ 741.099404] Lustre: 26464:0:(osd_handler.c:2123:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 741.113993] Lustre: 26464:0:(osd_handler.c:2123:osd_trans_dump_creds()) Skipped 1 previous similar message [ 744.304681] Lustre: 25162:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 744.317824] Lustre: 25162:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 1 previous similar message [ 744.325121] Lustre: 25162:0:(osd_handler.c:2092:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 744.333305] Lustre: 25162:0:(osd_handler.c:2092:osd_trans_dump_creds()) Skipped 1 previous similar message [ 744.341187] Lustre: 25162:0:(osd_handler.c:2099:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 744.348652] Lustre: 25162:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 1 previous similar message [ 744.355459] Lustre: 25162:0:(osd_handler.c:2109:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 744.363575] Lustre: 25162:0:(osd_handler.c:2109:osd_trans_dump_creds()) Skipped 1 previous similar message [ 744.373017] Lustre: 25162:0:(osd_handler.c:2116:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 744.385590] Lustre: 25162:0:(osd_handler.c:2116:osd_trans_dump_creds()) Skipped 1 previous similar message [ 744.397689] Lustre: 25162:0:(osd_handler.c:2123:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 744.417316] Lustre: 25162:0:(osd_handler.c:2123:osd_trans_dump_creds()) Skipped 1 previous similar message [ 749.277520] Lustre: 26464:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 749.288731] Lustre: 26464:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 1 previous similar message [ 749.295087] Lustre: 26464:0:(osd_handler.c:2092:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 749.301160] Lustre: 26464:0:(osd_handler.c:2092:osd_trans_dump_creds()) Skipped 1 previous similar message [ 749.307396] Lustre: 26464:0:(osd_handler.c:2099:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 749.313153] Lustre: 26464:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 1 previous similar message [ 749.319828] Lustre: 26464:0:(osd_handler.c:2109:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 749.330741] Lustre: 26464:0:(osd_handler.c:2109:osd_trans_dump_creds()) Skipped 1 previous similar message [ 749.338916] Lustre: 26464:0:(osd_handler.c:2116:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 749.359729] Lustre: 26464:0:(osd_handler.c:2116:osd_trans_dump_creds()) Skipped 1 previous similar message [ 749.364835] Lustre: 26464:0:(osd_handler.c:2123:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 749.372946] Lustre: 26464:0:(osd_handler.c:2123:osd_trans_dump_creds()) Skipped 1 previous similar message [ 757.499910] Lustre: 26464:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 757.508331] Lustre: 26464:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 3 previous similar messages [ 757.514790] Lustre: 26464:0:(osd_handler.c:2092:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 757.519781] Lustre: 26464:0:(osd_handler.c:2092:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 757.534501] Lustre: 26464:0:(osd_handler.c:2099:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 757.546448] Lustre: 26464:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 757.556876] Lustre: 26464:0:(osd_handler.c:2109:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 757.562780] Lustre: 26464:0:(osd_handler.c:2109:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 757.573831] Lustre: 26464:0:(osd_handler.c:2116:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 757.581777] Lustre: 26464:0:(osd_handler.c:2116:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 757.588075] Lustre: 26464:0:(osd_handler.c:2123:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 757.594000] Lustre: 26464:0:(osd_handler.c:2123:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 810.464648] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 810.477710] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 815.586225] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 816.610911] Lustre: server umount lustre-OST07c6 complete [ 826.535363] Lustre: server umount lustre-MDT0000 complete [ 843.134421] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 845.785572] Key type lgssc unregistered [ 846.062776] LNet: 28731:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 846.078104] LNetError: 28731:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 846.091309] LNet: Removed LNI 192.168.203.160@tcp [ 847.053668] Key type .llcrypt unregistered [ 847.056805] Key type ._llcrypt unregistered [ 864.934883] Key type ._llcrypt registered [ 864.938220] Key type .llcrypt registered [ 865.053213] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 866.117456] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 866.248153] alg: No test for adler32 (adler32-zlib) [ 867.481475] Lustre: Lustre: Build Version: 2.17.57_104_g6cc0f2c [ 867.760898] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 869.487170] Key type lgssc registered [ 870.615520] Lustre: Echo OBD driver; http://www.lustre.org/ [ 878.440611] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 878.469260] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 880.040098] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 885.024472] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 889.322191] LustreError: 30000:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 889.335892] LustreError: 30000:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 893.617850] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 894.087089] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 898.401782] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 906.347337] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 913.110273] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 920.707478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 921.130060] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 924.236887] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 928.027874] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 936.561498] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 953.836318] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 953.839532] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 953.862031] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 954.849497] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 954.867306] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 954.883200] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 958.954140] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 958.962907] Lustre: Skipped 1 previous similar message [ 960.083266] Lustre: server umount lustre-OST0000 complete [ 964.065119] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 964.068933] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 964.078241] Lustre: Skipped 1 previous similar message [ 964.085293] Lustre: Skipped 1 previous similar message [ 969.185762] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 969.194399] Lustre: Skipped 1 previous similar message [ 978.401219] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 978.663565] Lustre: server umount lustre-MDT0000 complete [ 979.424382] LustreError: 30000:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 979.443929] LustreError: 30000:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 982.516982] LustreError: 31846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788330432 with bad export cookie 9652039533850724927 [ 982.536132] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 982.538696] LustreError: 31846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 982.845427] Lustre: server umount lustre-MDT0001 complete [ 994.486064] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 997.459882] Key type lgssc unregistered [ 997.785502] LNet: 33739:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 997.794283] LNetError: 33739:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 997.810374] LNet: Removed LNI 192.168.203.160@tcp [ 998.831801] Key type .llcrypt unregistered [ 998.837880] Key type ._llcrypt unregistered [ 1016.768253] Key type ._llcrypt registered [ 1016.770817] Key type .llcrypt registered [ 1016.852560] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 1018.154549] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1018.178547] alg: No test for adler32 (adler32-zlib) [ 1019.250949] Lustre: Lustre: Build Version: 2.17.57_104_g6cc0f2c [ 1019.478772] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 1021.192412] Key type lgssc registered [ 1022.300982] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1029.553755] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1029.571446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1031.071789] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1035.909842] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 1040.364494] LustreError: 35000:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1040.399760] LustreError: 35000:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1044.536969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1045.006975] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1049.902441] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 1058.116238] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1064.435690] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1070.728505] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1071.071937] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1075.218291] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1076.800391] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 1083.717236] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1099.750930] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1099.764664] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1099.775710] Lustre: Skipped 1 previous similar message [ 1099.782931] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1104.268021] Lustre: server umount lustre-OST0000 complete [ 1109.985923] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1109.998377] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1110.009580] Lustre: Skipped 2 previous similar messages [ 1114.648762] LustreError: 37802:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (00000000c45e5147) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 1115.113499] LustreError: 34994:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1115.140827] LustreError: 34994:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1116.743287] Lustre: server umount lustre-MDT0000 complete [ 1119.203100] LustreError: 35728:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1119.224074] LustreError: 35728:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1120.467942] LustreError: 36843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788330570 with bad export cookie 4622048479589307689 [ 1120.475283] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1120.481211] LustreError: 36843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1120.925774] Lustre: server umount lustre-MDT0001 complete [ 1132.967799] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 1135.589682] Key type lgssc unregistered [ 1135.982339] LNet: 38553:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1135.992994] LNetError: 38553:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1136.022818] LNet: Removed LNI 192.168.203.160@tcp [ 1136.873269] Key type .llcrypt unregistered [ 1136.875107] Key type ._llcrypt unregistered [ 1165.717624] Key type ._llcrypt registered [ 1165.720699] Key type .llcrypt registered [ 1165.879421] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 1167.236768] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1167.277288] alg: No test for adler32 (adler32-zlib) [ 1168.486734] Lustre: Lustre: Build Version: 2.17.57_104_g6cc0f2c [ 1168.891368] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 1170.591173] Key type lgssc registered [ 1172.221861] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1180.982618] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1181.007523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1182.785344] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1188.154791] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 1192.936899] LustreError: 39966:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1192.957617] LustreError: 39966:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1197.899941] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1198.053533] LustreError: 39967:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1198.397751] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1203.378526] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 1211.961459] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1218.543512] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1225.433574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1225.799703] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1227.829670] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1232.267102] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 1239.981262] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1249.760538] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1249.784948] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1249.812601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1252.057396] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.60@tcp (stopping) [ 1253.345080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1253.349324] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1255.413438] Lustre: server umount lustre-MDT0000 complete [ 1257.140820] LustreError: 39962:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1260.451209] LustreError: 41808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788330710 with bad export cookie 5484635714376085275 [ 1260.468378] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1260.473784] LustreError: 41808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1262.262264] LustreError: 39962:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1262.320705] LustreError: 39962:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1262.336826] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.60@tcp (stopping) [ 1262.355803] Lustre: Skipped 2 previous similar messages [ 1263.589242] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1263.621287] Lustre: Skipped 1 previous similar message [ 1267.091951] Lustre: server umount lustre-MDT0001 complete [ 1282.512681] Lustre: server umount lustre-OST0000 complete [ 1299.092755] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 1301.618346] Key type lgssc unregistered [ 1301.938887] LNet: 43826:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1301.950034] LNetError: 43826:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1301.988566] LNet: Removed LNI 192.168.203.160@tcp [ 1302.960646] Key type .llcrypt unregistered [ 1302.963695] Key type ._llcrypt unregistered [ 1326.850317] Key type ._llcrypt registered [ 1326.853855] Key type .llcrypt registered [ 1326.987664] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 1328.301567] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1328.354847] alg: No test for adler32 (adler32-zlib) [ 1329.697820] Lustre: Lustre: Build Version: 2.17.57_104_g6cc0f2c [ 1329.944872] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 1331.671172] Key type lgssc registered [ 1332.858114] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1341.305660] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1341.326636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1342.726810] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1346.575384] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 1352.162931] LustreError: 45241:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1352.183835] LustreError: 45241:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1354.990209] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1355.434417] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1359.764719] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 1367.412082] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1372.739136] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1378.561409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1378.858402] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1385.032738] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 1388.105198] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1393.399841] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1400.032052] Lustre: lustre-OST0000: Not available for connect from 192.168.203.60@tcp (stopping) [ 1401.312452] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1401.324310] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1401.343123] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1403.372635] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1403.377387] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1403.379898] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1403.418426] Lustre: Skipped 1 previous similar message [ 1405.229368] Lustre: server umount lustre-OST0000 complete [ 1423.842090] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1423.855416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1423.864217] Lustre: Skipped 2 previous similar messages [ 1428.990328] LustreError: 45240:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1429.025618] LustreError: 45240:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1429.275369] Lustre: server umount lustre-MDT0000 complete [ 1432.911871] LustreError: 45222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788330882 with bad export cookie 3143511728954094287 [ 1432.915048] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1432.922330] LustreError: 45222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1447.391180] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1447.701369] Lustre: server umount lustre-MDT0001 complete [ 1459.406920] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 1462.758341] Key type lgssc unregistered [ 1463.007463] LNet: 48951:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1463.017297] LNetError: 48951:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1463.029056] LNet: Removed LNI 192.168.203.160@tcp [ 1463.973401] Key type .llcrypt unregistered [ 1463.975143] Key type ._llcrypt unregistered [ 1489.975691] Key type ._llcrypt registered [ 1489.987904] Key type .llcrypt registered [ 1490.162945] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 1491.126698] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1491.295674] alg: No test for adler32 (adler32-zlib) [ 1492.825772] Lustre: Lustre: Build Version: 2.17.57_104_g6cc0f2c [ 1493.178645] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 1494.903222] Key type lgssc registered [ 1495.825745] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1502.896939] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1502.942247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1504.461130] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1509.497980] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 1514.982938] LustreError: 50363:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1514.999531] LustreError: 50363:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1517.892337] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1518.266544] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1522.492245] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 1529.982816] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1536.026247] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1542.967683] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1543.388870] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1549.555386] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1549.748649] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 1556.605884] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1570.544324] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1573.966745] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1586.319172] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-02 2h37m14s [ 1588.032249] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1590.243056] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1590.249797] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1590.276416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1591.332816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1591.344046] Lustre: Skipped 4 previous similar messages [ 1594.059829] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.60@tcp (stopping) [ 1596.455823] Lustre: server umount lustre-MDT0000 complete [ 1597.415622] LustreError: 53081:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1597.434960] LustreError: 53081:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1599.598744] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1601.211985] LustreError: 50359:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1603.563085] LustreError: 51416:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1603.583149] LustreError: 51416:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1606.270458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1606.445522] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1606.643665] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1606.707337] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1607.670071] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1610.820592] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 1613.866340] Lustre: DEBUG MARKER: Wait for df (36657) ... [ 1615.459249] Lustre: DEBUG MARKER: done [ 1620.456945] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1620.460946] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1620.470457] Lustre: Skipped 2 previous similar messages [ 1620.479071] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1620.484663] Lustre: Skipped 1 previous similar message [ 1621.991463] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1622.006521] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1628.130832] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1628.134987] Lustre: Skipped 3 previous similar messages [ 1634.271245] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1634.449574] Lustre: server umount lustre-OST0000 complete [ 1638.368220] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1638.375959] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1638.383223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1638.387402] Lustre: Skipped 1 previous similar message [ 1638.398535] Lustre: Skipped 3 previous similar messages [ 1644.948153] Lustre: server umount lustre-MDT0000 complete [ 1649.507035] LustreError: 53010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788331099 with bad export cookie 15295438176459903951 [ 1649.516301] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1649.520354] LustreError: 53010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1650.002208] Lustre: server umount lustre-MDT0001 complete [ 1660.507590] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 1663.082948] Key type lgssc unregistered [ 1663.339128] LNet: 55306:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1663.344420] LNetError: 55306:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1663.354236] LNet: Removed LNI 192.168.203.160@tcp [ 1664.055503] Key type .llcrypt unregistered [ 1664.062957] Key type ._llcrypt unregistered [ 1708.616811] Key type ._llcrypt registered [ 1708.618958] Key type .llcrypt registered [ 1708.775621] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 1710.156907] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1710.264089] alg: No test for adler32 (adler32-zlib) [ 1711.397341] Lustre: Lustre: Build Version: 2.17.57_104_g6cc0f2c [ 1711.752828] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 1713.575176] Key type lgssc registered [ 1714.881264] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1723.781766] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1723.813427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1725.225634] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1725.245419] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1725.355161] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1729.939604] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 1738.825829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1738.976815] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1739.040236] Lustre: 57514:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1739.348282] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1744.806834] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 1753.674300] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1759.894928] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1767.691808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1767.892482] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1768.124407] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1774.238214] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1774.944126] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 1783.115615] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1790.600099] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-02 2h40m39s [ 1792.121583] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1798.824177] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1801.460916] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1801.532741] LustreError: 59059:0:(mdt_handler.c:2345:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1821.910133] Lustre: lustre-MDT0000: Client 3747c5e5-43c2-45f9-acfe-08684366cb99 (at 192.168.203.60@tcp) reconnecting [ 1841.559133] LustreError: 59059:0:(mdt_handler.c:2345:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1844.046760] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1846.780635] Lustre: DEBUG MARKER: done [ 1855.977262] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1855.995386] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1856.013170] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1856.493352] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1856.497447] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1856.538449] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1856.547236] Lustre: Skipped 1 previous similar message [ 1859.398536] Lustre: server umount lustre-OST0000 complete [ 1866.735650] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1866.741912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1866.756327] Lustre: Skipped 1 previous similar message [ 1870.083430] Lustre: server umount lustre-MDT0000 complete [ 1873.898823] LustreError: 58255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788331323 with bad export cookie 9062843585421874995 [ 1873.908737] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1873.915120] LustreError: 58255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1874.300856] Lustre: server umount lustre-MDT0001 complete [ 1889.496973] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 1892.943380] Key type lgssc unregistered [ 1893.506755] LNet: 61843:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1893.515898] LNetError: 61843:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1894.565889] LNet: Removed LNI 192.168.203.160@tcp [ 1896.123083] Key type .llcrypt unregistered [ 1896.127064] Key type ._llcrypt unregistered [ 1952.651960] Key type ._llcrypt registered [ 1952.664018] Key type .llcrypt registered [ 1952.804287] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 1954.834274] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1954.899450] alg: No test for adler32 (adler32-zlib) [ 1956.118974] Lustre: Lustre: Build Version: 2.17.57_104_g6cc0f2c [ 1956.586563] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 1958.463189] Key type lgssc registered [ 1959.795441] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1964.747703] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1964.761673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1965.992174] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1966.006552] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1966.179437] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1967.129062] Lustre: Failing over lustre-MDT0000 [ 1967.382692] Lustre: server umount lustre-MDT0000 complete [ 1975.601206] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 02:43:44 (1788331424) [ 1982.848510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1983.409113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1988.328151] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 1997.667936] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1997.821694] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1997.894414] Lustre: 64640:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1998.301040] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2004.168964] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 2014.255806] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2022.166707] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2029.443702] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2029.734112] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2029.957764] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2036.851024] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 2039.515210] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2045.362825] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2051.341410] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2059.744224] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2059.747679] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2059.764414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2060.257567] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2060.266753] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2060.304216] Lustre: Skipped 1 previous similar message [ 2064.582929] Lustre: server umount lustre-MDT0000 complete [ 2068.471974] LustreError: 64635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788331518 with bad export cookie 5037216015249893790 [ 2068.478355] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2068.493416] LustreError: 64635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2068.789482] Lustre: server umount lustre-MDT0001 complete [ 2072.030460] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2077.118930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2086.879926] Lustre: 63561:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788331520/real 1788331520] req@0000000082f2d52d x1875201392419840/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788331536 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2086.909504] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2086.926517] Lustre: Skipped 1 previous similar message [ 2087.239227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2090.015843] Lustre: 63561:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788331523/real 1788331523] req@00000000fa2f748e x1875201392419968/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788331539 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2091.999165] Lustre: 63562:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788331525/real 1788331525] req@0000000089908bfd x1875201392420096/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788331541 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2093.111487] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x45e7c9470de0f281 [ 2093.136428] Lustre: MGC192.168.203.160@tcp: Connection restored to 0@lo (at 0@lo) [ 2093.521665] LustreError: 66489:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2093.562690] LustreError: 66489:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2093.684627] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2093.838669] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2094.625967] Lustre: 63563:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788331528/real 1788331528] req@00000000fe278244 x1875201392420352/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788331544 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2096.673706] LustreError: 66490:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2097.704218] LustreError: 68413:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2097.741345] LustreError: 68413:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2099.710536] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 2099.743110] Lustre: 63560:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788331533/real 1788331533] req@00000000fd81cb6f x1875201392420480/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788331549 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2102.754090] LustreError: 68415:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2102.790841] LustreError: 68415:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2107.875255] LustreError: 66490:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2107.882737] LustreError: 66490:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2108.186589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2108.776032] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2114.046774] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2114.102146] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 2115.124761] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2122.264843] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2127.803202] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2134.497102] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2134.501206] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2134.531934] LustreError: 66490:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2134.749496] Lustre: server umount lustre-MDT0000 complete [ 2138.334467] LustreError: 66495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788331588 with bad export cookie 5037216015249896065 [ 2138.339242] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2138.353600] LustreError: 66495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2138.689078] Lustre: server umount lustre-MDT0001 complete [ 2145.083631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2154.180504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2157.023444] Lustre: 63560:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788331590/real 1788331590] req@00000000186b3630 x1875201392447232/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788331606 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2157.055694] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2157.077485] Lustre: Skipped 2 previous similar messages [ 2163.235316] LustreError: 63559:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000b8c5ff3d x1875201392448384/t0(0) o250->MGC192.168.203.160@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2163.720104] LustreError: 66490:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2163.756153] LustreError: 66490:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2163.824347] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2163.881275] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2167.991858] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 2175.466934] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2175.675567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2180.758717] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 2181.104191] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2188.038674] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2194.435311] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2206.689718] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2206.695686] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2206.703529] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2206.705557] Lustre: Skipped 1 previous similar message [ 2206.736554] Lustre: Skipped 2 previous similar messages [ 2208.706596] Lustre: server umount lustre-MDT0000 complete [ 2211.810490] LustreError: 71801:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2211.827353] LustreError: 71801:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 2212.986826] LustreError: 64635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788331662 with bad export cookie 5037216015249897444 [ 2212.996117] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2213.343571] Lustre: server umount lustre-MDT0001 complete [ 2219.720350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2222.902225] Lustre: DEBUG MARKER: files compared the same [ 2235.344149] Lustre: server umount lustre-OST0000 complete [ 2253.340836] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 2255.864868] Key type lgssc unregistered [ 2256.238563] LNet: 74370:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2256.248698] LNetError: 74370:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2256.264604] LNet: Removed LNI 192.168.203.160@tcp [ 2257.326408] Key type .llcrypt unregistered [ 2257.328640] Key type ._llcrypt unregistered [ 2285.460170] Key type ._llcrypt registered [ 2285.464253] Key type .llcrypt registered [ 2285.656928] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 2287.201015] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2287.284777] alg: No test for adler32 (adler32-zlib) [ 2288.494669] Lustre: Lustre: Build Version: 2.17.57_104_g6cc0f2c [ 2288.744204] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 2290.506986] Key type lgssc registered [ 2292.029909] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2301.161714] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2301.190786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2302.701599] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2307.354252] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug +malloc all [ 2312.164925] LustreError: 75783:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2312.185660] LustreError: 75783:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2316.310335] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2316.688943] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2321.814507] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug +malloc all [ 2329.450233] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2335.246251] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2343.074201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2343.579638] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2347.698544] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2350.767859] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug +malloc all [ 2359.626460] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2372.576632] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2372.579776] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2372.603438] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2372.630189] Lustre: Skipped 1 previous similar message [ 2372.646776] Lustre: Skipped 1 previous similar message [ 2377.700374] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2377.716624] Lustre: Skipped 1 previous similar message [ 2382.820305] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2382.833456] Lustre: Skipped 1 previous similar message [ 2383.328836] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2383.644840] Lustre: server umount lustre-OST0000 complete [ 2388.447910] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2388.451470] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2388.471084] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2393.055916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2393.056472] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2394.720191] Lustre: server umount lustre-MDT0000 complete [ 2398.177604] LustreError: 75784:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2398.194705] LustreError: 75784:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2399.228883] LustreError: 75764:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788331849 with bad export cookie 5735461969357474576 [ 2399.235301] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2399.244803] LustreError: 75764:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2399.793578] Lustre: server umount lustre-MDT0001 complete [ 2411.992905] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 2416.052176] Key type lgssc unregistered [ 2416.378224] LNet: 79318:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2416.389361] LNetError: 79318:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2416.414484] LNet: Removed LNI 192.168.203.160@tcp [ 2417.505210] Key type .llcrypt unregistered [ 2417.508858] Key type ._llcrypt unregistered [ 2445.200568] Key type ._llcrypt registered [ 2445.203334] Key type .llcrypt registered [ 2445.349345] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 2447.337817] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2447.376385] alg: No test for adler32 (adler32-zlib) [ 2448.620888] Lustre: Lustre: Build Version: 2.17.57_104_g6cc0f2c [ 2448.976423] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 2450.752719] Key type lgssc registered [ 2452.703957] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2462.191991] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2462.226561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2477.919339] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2483.039739] LustreError: 80707:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.160@tcp: failed processing log, type 4: rc = -110 [ 2509.791312] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2516.549923] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 2525.873469] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2536.558019] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2536.799806] Lustre: *** cfs_fail_loc=706, val=0*** [ 2537.257179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2537.379619] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2542.563312] LustreError: 80732:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2542.599060] LustreError: 80732:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2542.949826] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 2544.620480] LustreError: 81793:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2549.739682] LustreError: 81793:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2549.758577] LustreError: 81793:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2553.335046] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2553.931354] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2559.724341] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 2569.116076] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2575.875966] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2578.401028] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2578.411364] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2578.420213] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2579.433905] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2579.445443] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2579.458116] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2579.459405] Lustre: Skipped 1 previous similar message [ 2584.551683] LustreError: 80724:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2584.564066] LustreError: 80724:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2584.739479] Lustre: server umount lustre-OST0000 complete [ 2589.673204] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2589.677418] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2589.709773] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2589.722257] Lustre: Skipped 1 previous similar message [ 2594.785467] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2594.794459] Lustre: Skipped 1 previous similar message [ 2595.321238] Lustre: server umount lustre-MDT0000 complete [ 2599.245359] LustreError: 80713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788332049 with bad export cookie 5488102875145777480 [ 2599.253965] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2599.255744] LustreError: 80713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2599.550791] Lustre: server umount lustre-MDT0001 complete [ 2611.735582] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 2614.897557] Key type lgssc unregistered [ 2615.231697] LNet: 84325:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2615.246724] LNetError: 84325:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2615.264765] LNet: Removed LNI 192.168.203.160@tcp [ 2616.251767] Key type .llcrypt unregistered [ 2616.255210] Key type ._llcrypt unregistered [ 2643.351598] Key type ._llcrypt registered [ 2643.356958] Key type .llcrypt registered [ 2643.517477] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 2644.546042] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2644.748543] alg: No test for adler32 (adler32-zlib) [ 2645.863691] Lustre: Lustre: Build Version: 2.17.57_104_g6cc0f2c [ 2646.109127] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 2647.831119] Key type lgssc registered [ 2648.877829] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2657.152880] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2657.199549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2663.818398] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 2671.936896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2672.537565] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2677.276045] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 2677.735084] LustreError: 86425:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2677.762655] LustreError: 86425:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2682.852258] LustreError: 86424:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2682.879070] LustreError: 86424:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2685.761370] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2685.965466] LustreError: 86425:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2685.990062] LustreError: 86425:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2686.012992] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2691.053040] LustreError: 86429:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2691.072670] LustreError: 86429:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2692.364938] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 2696.163128] LustreError: 86428:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2696.176697] LustreError: 86428:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2700.197954] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2700.218174] Lustre: MGS: Client be8d9a0a-ce17-452f-89ae-59100bb00625 (at 0@lo) reconnecting [ 2700.253240] LustreError: 86439:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2700.462400] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2700.542077] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2705.580939] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 2721.249859] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2721.266325] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2721.288776] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2721.304530] Lustre: Skipped 1 previous similar message [ 2721.321658] Lustre: Skipped 1 previous similar message [ 2726.316933] Lustre: server umount lustre-OST0000 complete [ 2731.492721] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2731.496720] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2731.497313] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2731.511450] Lustre: Skipped 1 previous similar message [ 2731.539624] Lustre: Skipped 1 previous similar message [ 2736.418199] Lustre: server umount lustre-MDT0000 complete [ 2736.615940] LustreError: 86429:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2736.636838] LustreError: 86429:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2741.569125] Lustre: server umount lustre-MDT0001 complete [ 2745.568888] Lustre: server umount MGS complete [ 2758.209537] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 02:56:47 (1788332207) [ 2787.366161] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_hostid [ 2796.360430] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 2834.737263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2839.695482] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 2850.163525] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2850.383592] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2850.458862] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2850.562551] Lustre: lustre-MDT0001: new disk, initializing [ 2850.689429] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2850.727201] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2850.737230] Lustre: 93611:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2850.751321] LustreError: 93611:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2855.757324] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 2867.005640] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2867.263633] Lustre: lustre-OST0000: new disk, initializing [ 2867.273027] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2867.281524] Lustre: 94376:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2867.355570] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2874.163454] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 2875.489326] Lustre: 94745:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000005e8d877d x1875202115935744/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788332341 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2885.304943] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2892.504397] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2892.536838] Lustre: MGS: Client 1119ed32-3a4d-4ec2-8df2-734cc9981cdd (at 0@lo) reconnecting [ 2892.593609] Lustre: 95426:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2892.612356] LustreError: 93596:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2892.652173] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2892.740988] Lustre: lustre-MDT0000: new disk, initializing [ 2892.905385] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2892.931341] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2892.955965] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2893.988475] Lustre: 95451:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000379dd1b5 x1875202115942784/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788332359 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2897.403908] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 2899.108624] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2903.018339] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2903.026745] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2903.158564] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 2928.097418] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2928.108834] Lustre: Skipped 1 previous similar message [ 2928.109922] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2928.118939] Lustre: Skipped 1 previous similar message [ 2932.516215] Lustre: server umount lustre-OST0000 complete [ 2938.337417] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2938.338392] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2938.346210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2938.372807] Lustre: Skipped 1 previous similar message [ 2942.704203] Lustre: server umount lustre-MDT0000 complete [ 2943.460412] LustreError: 93585:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2943.479814] LustreError: 93585:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2952.576446] Lustre: server umount lustre-MDT0001 complete [ 2956.129865] Lustre: server umount MGS complete [ 2966.416622] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 03:00:15 (1788332415) [ 2982.614586] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 2985.119627] Key type lgssc unregistered [ 2985.355948] LNet: 97951:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2985.363870] LNetError: 97951:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2985.390739] LNet: Removed LNI 192.168.203.160@tcp [ 2986.166583] Key type .llcrypt unregistered [ 2986.170575] Key type ._llcrypt unregistered [ 3001.674410] Key type ._llcrypt registered [ 3001.678361] Key type .llcrypt registered [ 3001.788306] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 3002.666142] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3002.686399] alg: No test for adler32 (adler32-zlib) [ 3003.722983] Lustre: Lustre: Build Version: 2.17.57_104_g6cc0f2c [ 3003.922312] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 3005.728334] Key type lgssc registered [ 3006.876289] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3016.062760] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3016.103645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3017.668603] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3023.385938] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3027.938237] LustreError: 99180:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3027.977111] LustreError: 99180:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3032.853974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3033.057212] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3033.376829] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3038.190417] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3046.088799] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3052.600855] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3059.396706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3059.736950] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3062.839427] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 3066.521084] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3074.047518] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3088.353261] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3088.370882] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3088.395574] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3091.515822] Lustre: server umount lustre-OST0000 complete [ 3098.592477] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3098.597266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3098.617614] Lustre: Skipped 2 previous similar messages [ 3098.633421] Lustre: Skipped 2 previous similar messages [ 3101.665501] Lustre: server umount lustre-MDT0000 complete [ 3103.716828] LustreError: 99177:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3103.749423] LustreError: 99177:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3105.926897] LustreError: 101960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788332555 with bad export cookie 17808891619434623243 [ 3105.939308] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3105.942544] LustreError: 101960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3106.387797] Lustre: server umount lustre-MDT0001 complete [ 3119.363418] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 3122.487859] Key type lgssc unregistered [ 3122.985737] LNet: 102711:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3122.998558] LNetError: 102711:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3123.018757] LNet: Removed LNI 192.168.203.160@tcp [ 3124.077173] Key type .llcrypt unregistered [ 3124.080803] Key type ._llcrypt unregistered [ 3143.909228] Key type ._llcrypt registered [ 3143.911222] Key type .llcrypt registered [ 3143.989623] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 3145.366277] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3145.443719] alg: No test for adler32 (adler32-zlib) [ 3146.622735] Lustre: Lustre: Build Version: 2.17.57_104_g6cc0f2c [ 3147.066437] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 3148.839485] Key type lgssc registered [ 3149.996262] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3155.678586] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3155.683486] LustreError: 103792:0:(tgt_mount.c:2570:server_fill_super()) cfs_race id 716 sleeping [ 3155.690165] Lustre: Skipped 1 previous similar message [ 3155.707617] LustreError: 103791:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 waking [ 3155.716482] LustreError: 103792:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 awake: rc=4996 [ 3155.728102] LustreError: 103792:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3155.728996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3155.734368] LustreError: 103792:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3157.561300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3167.170506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3167.362875] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3172.923214] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3180.120837] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing lsmod [ 3182.982385] LustreError: 104999:0:(tgt_mount.c:2570:server_fill_super()) cfs_race id 716 sleeping [ 3182.983189] LustreError: 104998:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 waking [ 3182.999349] LustreError: 104999:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 awake: rc=4995 [ 3183.009731] LustreError: 104999:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3183.014232] LustreError: 104999:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3183.020017] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3183.559814] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3184.579280] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 3188.191985] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3188.201665] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3188.208745] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3189.729325] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3189.732613] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3189.754427] Lustre: Skipped 1 previous similar message [ 3189.764173] Lustre: Skipped 2 previous similar messages [ 3192.217040] Lustre: server umount lustre-MDT0000 complete [ 3192.886626] LustreError: 103824:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3192.910568] LustreError: 103824:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3196.263891] LustreError: 103810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788332646 with bad export cookie 5851677343241549142 [ 3196.276735] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3196.286081] LustreError: 103810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3197.023882] LustreError: 104261:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000214dc113 x1875202641253248/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3197.137467] Lustre: server umount lustre-MDT0001 complete [ 3211.318933] Lustre: server umount lustre-OST0000 complete [ 3220.488794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3221.006840] LustreError: 106062:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3221.113991] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3226.400628] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3226.607487] LustreError: 106063:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3230.711463] LustreError: 106062:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3235.581868] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3235.809085] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3235.987868] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3240.772238] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3248.738449] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3255.013472] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3262.261306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3262.649828] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3265.710273] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:97) [ 3269.306505] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3277.468763] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3291.115365] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3291.126865] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3291.143693] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3296.658703] Lustre: server umount lustre-OST0000 complete [ 3301.861042] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3301.879123] Lustre: Skipped 1 previous similar message [ 3301.885452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3301.893961] Lustre: Skipped 2 previous similar messages [ 3306.881869] Lustre: server umount lustre-MDT0000 complete [ 3306.981260] LustreError: 106058:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3306.996379] LustreError: 106058:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3311.448207] LustreError: 106044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788332761 with bad export cookie 5851677343241550073 [ 3311.450149] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3311.458711] LustreError: 106044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3311.918244] Lustre: server umount lustre-MDT0001 complete [ 3324.418639] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 3327.223648] Key type lgssc unregistered [ 3327.470991] LNet: 109614:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3327.479262] LNetError: 109614:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3327.496836] LNet: Removed LNI 192.168.203.160@tcp [ 3328.416245] Key type .llcrypt unregistered [ 3328.419105] Key type ._llcrypt unregistered [ 3353.724545] Key type ._llcrypt registered [ 3353.728867] Key type .llcrypt registered [ 3353.878639] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 3355.021516] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3355.034670] alg: No test for adler32 (adler32-zlib) [ 3356.094054] Lustre: Lustre: Build Version: 2.17.57_104_g6cc0f2c [ 3356.298122] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 3357.959144] Key type lgssc registered [ 3359.204695] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3366.335989] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3366.363273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3367.828186] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3372.093353] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3377.128628] LustreError: 111028:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3377.144367] LustreError: 111028:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3380.284820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3380.605164] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3385.030229] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3392.965683] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3398.608916] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3405.154955] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3405.635606] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3412.582708] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3412.814798] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 3420.433049] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3429.708829] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3432.729659] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3438.562471] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3438.581228] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3438.611571] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3440.639099] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3440.648806] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3440.651316] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3440.690121] Lustre: Skipped 1 previous similar message [ 3445.735031] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3445.743825] Lustre: Skipped 1 previous similar message [ 3450.854251] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3450.865331] Lustre: Skipped 1 previous similar message [ 3451.360362] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3451.577548] Lustre: server umount lustre-OST0000 complete [ 3455.975218] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3455.983833] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3456.007356] Lustre: Skipped 1 previous similar message [ 3456.023773] Lustre: Skipped 1 previous similar message [ 3461.819383] Lustre: server umount lustre-MDT0000 complete [ 3466.071191] LustreError: 111008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788332916 with bad export cookie 2291647028980299610 [ 3466.072422] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3466.080093] LustreError: 111008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3466.342264] Lustre: server umount lustre-MDT0001 complete [ 3479.613166] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 3482.547159] Key type lgssc unregistered [ 3482.760134] LNet: 114667:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3482.766408] LNetError: 114667:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3482.783752] LNet: Removed LNI 192.168.203.160@tcp [ 3483.738290] Key type .llcrypt unregistered [ 3483.742148] Key type ._llcrypt unregistered [ 3498.990380] Key type ._llcrypt registered [ 3498.994337] Key type .llcrypt registered [ 3499.109127] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 3499.799581] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3499.936193] alg: No test for adler32 (adler32-zlib) [ 3501.023855] Lustre: Lustre: Build Version: 2.17.57_104_g6cc0f2c [ 3501.400465] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 3503.087188] Key type lgssc registered [ 3504.009533] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3510.255828] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3510.288178] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3511.636235] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3515.440332] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3522.018433] LustreError: 115899:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3522.030182] LustreError: 115899:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3523.034628] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3523.422644] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3528.287805] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3537.765788] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3544.648397] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3554.302527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3554.633190] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3556.719874] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 3561.192932] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3569.504918] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3579.364500] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3579.371122] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3579.374211] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3584.494690] LustreError: 117830:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3584.509520] LustreError: 117830:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3584.684272] Lustre: server umount lustre-OST0000 complete [ 3589.601704] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3589.604096] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3589.616822] Lustre: Skipped 2 previous similar messages [ 3589.620847] Lustre: Skipped 1 previous similar message [ 3594.720609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3594.740756] Lustre: Skipped 2 previous similar messages [ 3595.521443] Lustre: server umount lustre-MDT0000 complete [ 3599.842822] LustreError: 117489:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3599.862336] LustreError: 117489:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3599.951963] LustreError: 115882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788333049 with bad export cookie 10343372110077539951 [ 3599.960457] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3599.963852] LustreError: 115882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3600.509071] Lustre: server umount lustre-MDT0001 complete [ 3612.294104] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 3615.416237] Key type lgssc unregistered [ 3615.685649] LNet: 119432:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3615.707583] LNetError: 119432:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3615.737839] LNet: Removed LNI 192.168.203.160@tcp [ 3617.009787] Key type .llcrypt unregistered [ 3617.014686] Key type ._llcrypt unregistered [ 3640.938220] Key type ._llcrypt registered [ 3640.940092] Key type .llcrypt registered [ 3641.062914] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 3642.001603] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3642.057175] alg: No test for adler32 (adler32-zlib) [ 3643.219997] Lustre: Lustre: Build Version: 2.17.57_104_g6cc0f2c [ 3643.527756] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 3645.223240] Key type lgssc registered [ 3646.379507] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3653.912692] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3653.934755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3655.436781] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3659.619652] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3665.893887] LustreError: 120848:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3665.917094] LustreError: 120848:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3668.463373] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3668.953958] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3673.404782] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3681.039229] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3687.037963] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3693.638625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3694.105361] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3700.233284] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:163 to 0x240000401:193) [ 3701.673099] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3710.727678] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3721.288538] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3724.606884] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3724.611925] Lustre: Skipped 2 previous similar messages [ 3726.952439] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3727.019489] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3728.945149] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3728.955802] Lustre: Skipped 2 previous similar messages [ 3735.081760] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3735.084258] Lustre: Skipped 1 previous similar message [ 3741.363114] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.60@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3741.390875] Lustre: Skipped 2 previous similar messages [ 3748.490826] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.60@tcp 192.168.0.[2,10]@tcp [ 3754.079794] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.60/24@tcp in log lustre-MDT0000 [ 3754.084669] Lustre: Skipped 2 previous similar messages [ 3758.045940] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.60/24@tcp [ 3758.051679] Lustre: Skipped 1 previous similar message [ 3771.876157] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3771.887130] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3771.908683] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3774.944243] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3774.953146] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3774.956725] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3775.011066] Lustre: Skipped 1 previous similar message [ 3776.903566] Lustre: server umount lustre-OST0000 complete [ 3785.186738] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3785.188487] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3785.214077] Lustre: Skipped 1 previous similar message [ 3788.501348] Lustre: server umount lustre-MDT0000 complete [ 3790.305430] LustreError: 120843:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3790.340994] LustreError: 120843:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3792.419261] LustreError: 124635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788333242 with bad export cookie 16002434683784665059 [ 3792.421553] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3792.438823] LustreError: 124635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3792.757604] Lustre: server umount lustre-MDT0001 complete [ 3804.099790] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 3806.887591] Key type lgssc unregistered [ 3807.158734] LNet: 125840:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3807.174253] LNetError: 125840:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3807.193609] LNet: Removed LNI 192.168.203.160@tcp [ 3808.074711] Key type .llcrypt unregistered [ 3808.079799] Key type ._llcrypt unregistered [ 3839.595860] Key type ._llcrypt registered [ 3839.599449] Key type .llcrypt registered [ 3839.730994] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 3840.762192] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3840.883611] alg: No test for adler32 (adler32-zlib) [ 3841.943339] Lustre: Lustre: Build Version: 2.17.57_104_g6cc0f2c [ 3842.228459] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 3843.927307] Key type lgssc registered [ 3845.179639] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3853.227587] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3853.263988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3854.733430] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3854.761178] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.60/24@tcp [ 3854.762161] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3859.553087] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3865.056618] LustreError: 127401:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3865.086690] LustreError: 127401:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3869.384648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3869.840627] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3869.902762] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.60/24@tcp [ 3869.915988] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3875.037517] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3883.067546] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3891.146238] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3902.375924] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3903.168298] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3913.533918] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3921.110144] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3925.757541] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:201 to 0x240000401:225) [ 3936.231151] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3936.262808] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3936.266649] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3936.290533] Lustre: Skipped 1 previous similar message [ 3941.354730] LustreError: 129232:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3941.458548] Lustre: server umount lustre-OST0000 complete [ 3946.472163] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3946.479863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3946.495527] Lustre: Skipped 2 previous similar messages [ 3946.511074] Lustre: Skipped 1 previous similar message [ 3951.588698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3951.602603] Lustre: Skipped 1 previous similar message [ 3956.706382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3956.723454] Lustre: Skipped 1 previous similar message [ 3959.264768] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3959.424147] Lustre: server umount lustre-MDT0000 complete [ 3961.826488] LustreError: 127401:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3961.860611] LustreError: 127401:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3964.018667] LustreError: 127382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788333413 with bad export cookie 13129094876554827808 [ 3964.028129] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3964.040257] LustreError: 127382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3964.517827] Lustre: server umount lustre-MDT0001 complete [ 3980.854683] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 3984.583881] Key type lgssc unregistered [ 3985.040218] LNet: 130993:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3985.066423] LNetError: 130993:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3985.103731] LNet: Removed LNI 192.168.203.160@tcp [ 3985.975939] Key type .llcrypt unregistered [ 3985.978958] Key type ._llcrypt unregistered [ 4017.691667] Key type ._llcrypt registered [ 4017.696311] Key type .llcrypt registered [ 4017.862308] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_hostid [ 4037.140954] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 4038.908763] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4038.957546] alg: No test for adler32 (adler32-zlib) [ 4040.109921] Lustre: Lustre: Build Version: 2.17.57_104_g6cc0f2c [ 4040.247251] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 4041.887306] Key type lgssc registered [ 4043.459776] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4080.787699] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4080.831744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4082.019141] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4082.042446] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4082.100238] Lustre: lustre-MDT0000: new disk, initializing [ 4082.225961] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4082.253098] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4087.487330] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 4100.542397] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4100.627151] Lustre: 134393:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4100.767860] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4100.772598] Lustre: Skipped 1 previous similar message [ 4100.959463] Lustre: lustre-MDT0001: new disk, initializing [ 4101.150646] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4101.268972] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4101.294830] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4106.878915] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 4117.317357] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4123.980745] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4131.059446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4131.291766] Lustre: lustre-OST0000: new disk, initializing [ 4131.294644] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4131.298880] Lustre: 136362:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4131.382796] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4137.678973] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 4141.651212] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4141.667035] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4141.862559] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4148.254275] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4153.827277] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4154.174838] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4159.399262] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4159.625249] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4162.539338] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4162.551869] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4162.573548] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4162.598391] LustreError: Skipped 1 previous similar message [ 4167.664631] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4167.678924] Lustre: Skipped 1 previous similar message [ 4168.128750] Lustre: server umount lustre-OST0000 complete [ 4172.771202] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4172.772145] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4172.777587] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4172.777596] Lustre: Skipped 1 previous similar message [ 4172.825903] Lustre: Skipped 2 previous similar messages [ 4177.889758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4177.897325] Lustre: Skipped 2 previous similar messages [ 4178.441579] Lustre: server umount lustre-MDT0000 complete [ 4181.876657] LustreError: 134386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788333631 with bad export cookie 9909436468947809012 [ 4181.878947] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4181.886202] LustreError: 134386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4182.285972] Lustre: server umount lustre-MDT0001 complete [ 4191.790920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4192.257120] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4196.850357] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 4197.351287] LustreError: 138581:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4197.372566] LustreError: 138581:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4202.466806] LustreError: 138580:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4205.821464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4206.365309] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4211.620410] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 4221.305202] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4227.456711] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4235.182275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4235.449273] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4242.562734] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 4250.824251] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4255.363099] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4255.490435] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4261.132285] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4261.336754] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4273.134736] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4273.324643] Lustre: lustre-OST0001: new disk, initializing [ 4273.327826] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4273.331942] Lustre: 141950:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4273.410996] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4279.446788] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4279.454168] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4279.644234] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4281.555313] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 4291.104360] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4291.347660] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4296.044872] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4296.318707] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4303.334869] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9a8e05c18000.ost_server_uuid 50 [ 4305.154840] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9a8e05c18000.ost_server_uuid in IDLE state after 0 sec [ 4315.634659] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4315.642349] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4315.652991] LustreError: Skipped 1 previous similar message [ 4315.662993] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4315.697537] Lustre: Skipped 1 previous similar message [ 4321.125996] Lustre: server umount lustre-OST0001 complete [ 4325.858721] LustreError: 140402:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4325.876341] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4325.885144] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4325.885152] Lustre: Skipped 2 previous similar messages [ 4325.892192] LustreError: 140402:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4325.906259] Lustre: Skipped 2 previous similar messages [ 4331.006685] LustreError: 140400:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4331.034470] LustreError: 140400:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4331.824413] Lustre: server umount lustre-OST0000 complete [ 4336.099143] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4336.107750] Lustre: Skipped 1 previous similar message [ 4342.183420] Lustre: server umount lustre-MDT0000 complete [ 4346.341485] LustreError: 138581:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4346.391371] LustreError: 138581:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4348.296675] LustreError: 138562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788333798 with bad export cookie 9909436468947809894 [ 4348.319395] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4348.335485] LustreError: 138562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4348.980112] Lustre: server umount lustre-MDT0001 complete [ 4371.241467] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 4374.254775] Key type lgssc unregistered [ 4374.565841] LNet: 144953:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4374.579697] LNetError: 144953:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4375.654429] LNet: Removed LNI 192.168.203.160@tcp [ 4376.503299] Key type .llcrypt unregistered [ 4376.505213] Key type ._llcrypt unregistered [ 4418.833256] Key type ._llcrypt registered [ 4418.835733] Key type .llcrypt registered [ 4418.952749] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_hostid [ 4433.712930] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 4435.117238] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4435.182105] alg: No test for adler32 (adler32-zlib) [ 4436.309467] Lustre: Lustre: Build Version: 2.17.57_104_g6cc0f2c [ 4436.581251] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 4438.231275] Key type lgssc registered [ 4439.226779] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4472.395257] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4472.433768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4473.650131] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4473.688704] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4473.783149] Lustre: lustre-MDT0000: new disk, initializing [ 4473.882475] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4473.906139] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4477.960437] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 4488.161386] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4488.266867] Lustre: 149103:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4488.319261] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4488.325855] Lustre: Skipped 1 previous similar message [ 4488.411594] Lustre: lustre-MDT0001: new disk, initializing [ 4488.514525] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4488.564423] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4488.580094] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4493.319954] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 4502.768967] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4509.194750] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4516.419398] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4516.702316] Lustre: lustre-OST0000: new disk, initializing [ 4516.708106] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4516.724557] Lustre: 151071:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4516.830725] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4522.045666] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4522.067199] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4522.186438] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4522.879903] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 4531.841139] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4536.602929] Lustre: Failing over lustre-OST0000 [ 4536.837649] Lustre: server umount lustre-OST0000 complete [ 4553.168178] Lustre: 147328:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788333987/real 1788333987] req@0000000060467485 x1875203992879616/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788334003 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4553.184206] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4553.215220] Lustre: 147328:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4553.246522] Lustre: Skipped 1 previous similar message [ 4553.685340] Lustre: 147329:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788333987/real 1788333987] req@00000000cdb296f8 x1875203992880256/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788334003 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4556.386681] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4556.804965] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4556.836878] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4558.818416] Lustre: 147326:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788333992/real 1788333992] req@00000000700f6890 x1875203992881408/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788334008 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4558.866274] Lustre: 147326:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4563.403128] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 4563.949435] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4564.960065] Lustre: 147326:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788333998/real 1788333998] req@000000004e144523 x1875203992882560/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788334014 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4566.232552] Lustre: lustre-OST0000: Recovery over after 0:03, of 3 clients 3 recovered and 0 were evicted. [ 4566.233357] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4567.217915] Lustre: Failing over lustre-MDT0000 [ 4567.520486] Lustre: server umount lustre-MDT0000 complete [ 4569.063367] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4569.072065] Lustre: Skipped 2 previous similar messages [ 4574.188178] LustreError: 149114:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4574.208974] LustreError: 149114:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 4576.444785] LustreError: 149110:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4579.299903] LustreError: 149109:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4579.336243] LustreError: 149109:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4581.562257] LustreError: 149110:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4585.440116] Lustre: 147327:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788334019/real 1788334019] req@000000004e144523 x1875203992892416/t0(0) o400->MGC192.168.203.160@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788334035 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4585.477542] Lustre: 147327:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4585.500110] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4586.682857] LustreError: 149108:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4586.718142] LustreError: 149108:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 4587.660310] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4595.681020] LustreError: 147325:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000445bd84e x1875203992897920/t0(0) o250->MGC192.168.203.160@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4595.734049] LustreError: 150022:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4595.757960] LustreError: 150022:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 4596.092666] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4596.209636] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4597.938468] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4601.299383] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 4601.338235] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4601.351224] Lustre: Skipped 1 previous similar message [ 4601.375688] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4601.430790] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4611.572236] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4611.577553] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4611.594715] Lustre: Skipped 1 previous similar message [ 4615.071827] Lustre: server umount lustre-OST0000 complete [ 4621.793368] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4621.793749] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4621.802558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4621.802567] Lustre: Skipped 1 previous similar message [ 4621.828813] Lustre: Skipped 1 previous similar message [ 4625.151100] Lustre: server umount lustre-MDT0000 complete [ 4626.913884] LustreError: 149113:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4626.926993] LustreError: 149113:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4628.792243] LustreError: 149095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788334078 with bad export cookie 11692754635229306500 [ 4628.797558] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4628.808068] LustreError: 149095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4629.078825] Lustre: server umount lustre-MDT0001 complete [ 4639.201719] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 4641.938280] Key type lgssc unregistered [ 4642.224769] LNet: 154730:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4642.234597] LNetError: 154730:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4642.257617] LNet: Removed LNI 192.168.203.160@tcp [ 4643.057307] Key type .llcrypt unregistered [ 4643.059832] Key type ._llcrypt unregistered [ 4678.389685] Key type ._llcrypt registered [ 4678.396033] Key type .llcrypt registered [ 4678.505482] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 4679.577491] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4679.592948] alg: No test for adler32 (adler32-zlib) [ 4680.812808] Lustre: Lustre: Build Version: 2.17.57_104_g6cc0f2c [ 4681.084749] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 4682.784077] Key type lgssc registered [ 4683.997950] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4692.111777] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4692.134410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4693.704113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4698.658508] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 4703.204323] LustreError: 156264:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4703.228633] LustreError: 156264:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4708.327464] LustreError: 156263:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4708.655577] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4713.207034] LustreError: 156292:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4713.216963] LustreError: 156292:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4713.225293] LustreError: 156292:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 20, retries 0, failed: rc = -5 [ 4713.638752] Lustre: server umount lustre-MDT0000 complete [ 4730.341142] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_hostid [ 4740.289153] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 4777.310629] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4777.772708] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4777.828548] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4778.042925] Lustre: lustre-MDT0000: new disk, initializing [ 4778.200040] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4778.228068] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4783.205435] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 4792.663062] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4792.784818] Lustre: 159593:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4792.824330] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4792.827851] Lustre: Skipped 1 previous similar message [ 4792.889805] Lustre: lustre-MDT0001: new disk, initializing [ 4793.009277] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4793.052838] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4793.064502] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4797.702101] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 4807.510315] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4812.691658] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4818.941420] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4819.218278] Lustre: lustre-OST0000: new disk, initializing [ 4819.227066] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4819.233268] Lustre: 161561:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4819.291368] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4824.642140] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4824.657867] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4824.793626] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4825.669312] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 4834.176642] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4838.898345] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4840.526248] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 03:31:29 (1788334289) [ 5100.513371] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5100.523181] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5100.552928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5101.536145] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5101.536840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5101.560623] Lustre: Skipped 2 previous similar messages [ 5103.177989] Lustre: server umount lustre-MDT0000 complete [ 5106.665058] LustreError: 159603:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5106.684535] LustreError: 159603:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5107.637550] LustreError: 159586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788334557 with bad export cookie 13765712769496086571 [ 5107.643036] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5107.662967] LustreError: 159586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5111.783523] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5111.784777] LustreError: 159601:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5111.808895] Lustre: Skipped 1 previous similar message [ 5111.835233] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5113.962153] Lustre: server umount lustre-MDT0001 complete [ 5124.651696] Lustre: server umount lustre-OST0000 complete [ 5134.500049] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_hostid [ 5144.661830] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 5182.918722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5183.324413] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5183.363518] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5183.471478] Lustre: lustre-MDT0000: new disk, initializing [ 5183.562510] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5183.590589] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5188.236327] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 5199.363434] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5199.504504] Lustre: 166235:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5199.593735] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5199.596632] Lustre: Skipped 1 previous similar message [ 5199.681570] Lustre: lustre-MDT0001: new disk, initializing [ 5199.829430] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5199.885553] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5199.909867] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5204.740040] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 5214.516453] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5220.654372] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5227.274539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5227.564702] Lustre: lustre-OST0000: new disk, initializing [ 5227.568810] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5227.577916] Lustre: 168201:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5227.633676] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5233.774797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5233.796886] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5233.900109] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 5233.938263] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5242.957462] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5247.381690] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5247.578254] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5251.492368] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5251.659807] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5254.625590] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5254.632072] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5254.640103] LustreError: Skipped 1 previous similar message [ 5254.642601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5254.642612] Lustre: Skipped 1 previous similar message [ 5254.686351] Lustre: Skipped 1 previous similar message [ 5259.418493] Lustre: server umount lustre-OST0000 complete [ 5264.865551] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5264.869625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5264.881590] Lustre: Skipped 1 previous similar message [ 5264.890384] Lustre: Skipped 2 previous similar messages [ 5269.001534] Lustre: server umount lustre-MDT0000 complete [ 5269.987305] LustreError: 166247:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5270.008454] LustreError: 166247:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5272.453262] LustreError: 166227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788334722 with bad export cookie 13765712769496245394 [ 5272.458382] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5272.467028] LustreError: 166227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5272.759031] Lustre: server umount lustre-MDT0001 complete [ 5280.134868] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 03:38:49 (1788334729) [ 5295.352839] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_hostid [ 5305.094604] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 5338.813899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5339.021631] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5339.057621] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5339.126174] Lustre: lustre-MDT0000: new disk, initializing [ 5339.188204] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5339.205236] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5343.890855] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 5353.707752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5353.812925] Lustre: 172957:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5354.059284] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5357.989489] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 5366.541780] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5372.449413] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5379.048208] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5379.319746] Lustre: lustre-OST0000: new disk, initializing [ 5379.323952] Lustre: Skipped 1 previous similar message [ 5379.328863] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5379.338226] Lustre: Skipped 2 previous similar messages [ 5379.344708] Lustre: 174933:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5386.093095] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 5388.328234] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5388.335170] Lustre: Skipped 1 previous similar message [ 5388.350659] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5388.491305] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5395.481545] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5404.130139] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5404.136048] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5404.140299] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5404.149566] Lustre: Skipped 1 previous similar message [ 5404.165245] Lustre: Skipped 1 previous similar message [ 5409.250661] LustreError: 174913:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5409.285185] LustreError: 174913:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5409.366753] Lustre: server umount lustre-OST0000 complete [ 5419.538413] Lustre: server umount lustre-MDT0000 complete [ 5423.318962] LustreError: 172950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788334873 with bad export cookie 13765712769496246283 [ 5423.325802] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5423.336171] LustreError: 172950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5423.770843] Lustre: server umount lustre-MDT0001 complete [ 5433.143683] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 03:41:22 (1788334882) [ 5448.754699] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_hostid [ 5457.868681] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 5493.380035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5493.578555] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5493.602565] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5493.672888] Lustre: lustre-MDT0000: new disk, initializing [ 5493.740845] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5493.748303] Lustre: Skipped 2 previous similar messages [ 5493.760935] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5498.389396] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 5509.716783] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5509.908789] Lustre: 179256:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5510.212787] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5514.409179] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 5523.769855] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5529.515784] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5535.808060] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5536.092259] Lustre: 181233:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5542.470836] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5542.618917] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 5542.621128] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5551.919238] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5561.316056] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5561.317397] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5561.330243] Lustre: Skipped 3 previous similar messages [ 5561.340628] Lustre: Skipped 3 previous similar messages [ 5565.134568] Lustre: server umount lustre-OST0000 complete [ 5574.684150] Lustre: server umount lustre-MDT0000 complete [ 5576.674561] LustreError: 179268:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5576.696910] LustreError: 179268:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5578.074156] LustreError: 181232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788335028 with bad export cookie 13765712769496247368 [ 5578.077259] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5578.086613] LustreError: 181232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5578.415320] Lustre: server umount lustre-MDT0001 complete [ 5588.455166] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 5591.120721] Key type lgssc unregistered [ 5591.463648] LNet: 183087:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5591.472317] LNetError: 183087:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5591.497562] LNet: Removed LNI 192.168.203.160@tcp [ 5592.336635] Key type .llcrypt unregistered [ 5592.338833] Key type ._llcrypt unregistered [ 5614.343341] Key type ._llcrypt registered [ 5614.346354] Key type .llcrypt registered [ 5614.477723] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 5615.493731] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5615.669992] alg: No test for adler32 (adler32-zlib) [ 5616.737831] Lustre: Lustre: Build Version: 2.17.57_104_g6cc0f2c [ 5617.028615] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 5618.679151] Key type lgssc registered [ 5619.719491] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5626.825773] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5626.854416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5628.364270] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5632.563871] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 5638.625200] LustreError: 184492:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5638.637037] LustreError: 184492:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5640.586736] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5641.014668] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5645.600689] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 5653.166144] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5658.712864] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5665.307966] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5665.645439] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5671.378453] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 5673.850844] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5678.714345] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5697.507959] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5697.520896] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5697.562673] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5699.555991] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5699.556466] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5699.556778] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5702.779287] Lustre: server umount lustre-OST0000 complete [ 5707.744714] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5707.759936] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5707.793559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5707.809688] Lustre: Skipped 1 previous similar message [ 5709.794114] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5709.821297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5709.827249] Lustre: Skipped 1 previous similar message [ 5712.598868] Lustre: server umount lustre-MDT0000 complete [ 5714.920866] LustreError: 185221:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5714.943312] LustreError: 185221:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5716.396873] LustreError: 184473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788335166 with bad export cookie 12105041086549844894 [ 5716.399655] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5716.405874] LustreError: 184473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5716.695938] Lustre: server umount lustre-MDT0001 complete [ 5727.577510] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 5729.947878] Key type lgssc unregistered [ 5730.271789] LNet: 188048:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5730.288200] LNetError: 188048:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5730.299748] LNet: Removed LNI 192.168.203.160@tcp [ 5731.122407] Key type .llcrypt unregistered [ 5731.124783] Key type ._llcrypt unregistered [ 5753.607184] Key type ._llcrypt registered [ 5753.609647] Key type .llcrypt registered [ 5753.672650] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 5754.281978] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5754.319773] alg: No test for adler32 (adler32-zlib) [ 5755.440723] Lustre: Lustre: Build Version: 2.17.57_104_g6cc0f2c [ 5755.680201] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 5757.327234] Key type lgssc registered [ 5758.291576] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5764.868242] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5764.884450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5766.234907] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5770.254123] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 5776.355514] LustreError: 189476:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5776.384716] LustreError: 189476:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5777.597420] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5777.921662] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5781.696614] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 5788.252777] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5793.251691] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5799.817291] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5800.166676] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5804.985754] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 5810.971439] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5811.442893] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5817.826301] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5817.839112] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5817.849606] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5818.571164] Lustre: lustre-OST0000: Not available for connect from 192.168.203.60@tcp (stopping) [ 5821.929614] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5821.934567] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5821.944088] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5823.275720] Lustre: server umount lustre-OST0000 complete [ 5829.701270] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9a8e06372800.ost_server_uuid 50 [ 5831.260654] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5844.452704] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5844.466612] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5844.487290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5844.493799] Lustre: Skipped 1 previous similar message [ 5847.523512] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5848.554306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5848.560779] Lustre: Skipped 3 previous similar messages [ 5850.042318] Lustre: server umount lustre-MDT0000 complete [ 5853.391088] LustreError: 191319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788335303 with bad export cookie 14113712623379669527 [ 5853.404332] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5853.408058] LustreError: 191319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5853.647669] Lustre: server umount lustre-MDT0001 complete [ 5862.083816] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 03:48:31 (1788335311) [ 5867.787752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5868.140738] LustreError: 193388:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5868.186658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5871.667500] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 5873.633149] LustreError: 193389:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5878.761191] LustreError: 193388:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5878.837912] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5879.185579] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5882.974953] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 5889.820809] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5894.762135] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5901.178485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5901.525337] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5906.895371] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 5913.221739] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5913.847378] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5918.892888] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5919.020299] Lustre: lustre-OST0001: new disk, initializing [ 5919.022901] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5919.028704] Lustre: 196231:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5919.123135] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5924.179401] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 5924.932416] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5924.952371] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5925.011511] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5931.478601] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5934.560584] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5934.566837] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5934.581171] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5939.769131] Lustre: server umount lustre-OST0000 complete [ 5940.713997] LustreError: 195211:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5940.729574] LustreError: 195211:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5944.983990] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5945.247336] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5948.972606] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5949.132215] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5950.964878] LustreError: 195210:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5950.983322] LustreError: 195210:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 5966.315189] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5966.317540] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5966.323198] LustreError: Skipped 2 previous similar messages [ 5966.327814] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5966.336500] Lustre: Skipped 2 previous similar messages [ 5966.345109] Lustre: Skipped 3 previous similar messages [ 5968.283827] Lustre: server umount lustre-OST0001 complete [ 5977.619800] Lustre: server umount lustre-MDT0000 complete [ 5980.602497] LustreError: 194125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788335430 with bad export cookie 14113712623379670605 [ 5980.607067] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5980.614159] LustreError: 194125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5980.887361] Lustre: server umount lustre-MDT0001 complete [ 6001.395839] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 03:50:50 (1788335450) [ 6007.470389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6007.742603] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6007.798713] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6008.062776] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6012.571967] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 6020.246050] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6020.359795] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6020.445887] Lustre: 199669:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6024.629505] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 6030.717858] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6035.193979] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6040.689944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6040.902210] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6045.279511] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 6047.315319] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6050.659599] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6056.233028] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6056.320976] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6061.370426] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 6067.560856] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6072.803927] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6072.807254] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6072.814445] LustreError: Skipped 1 previous similar message [ 6072.826085] Lustre: Skipped 2 previous similar messages [ 6072.831485] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6072.831500] Lustre: Skipped 5 previous similar messages [ 6075.761801] Lustre: server umount lustre-OST0000 complete [ 6077.926040] LustreError: 201516:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6077.947494] LustreError: 201516:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 6093.288298] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6093.295741] LustreError: Skipped 1 previous similar message [ 6095.415805] Lustre: server umount lustre-OST0001 complete [ 6104.878540] Lustre: server umount lustre-MDT0000 complete [ 6107.329920] LustreError: 201536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788335557 with bad export cookie 14113712623379672012 [ 6107.333852] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6107.335914] LustreError: 201536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6107.553550] Lustre: server umount lustre-MDT0001 complete [ 6124.661918] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 03:52:54 (1788335574) [ 6136.804114] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_hostid [ 6143.919627] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 6169.155303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6169.315877] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6169.353496] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6169.450854] Lustre: lustre-MDT0000: new disk, initializing [ 6169.495564] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6169.502251] Lustre: Skipped 3 previous similar messages [ 6169.518363] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6172.443851] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 6180.414148] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6180.512046] Lustre: 207758:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6180.542196] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6180.545101] Lustre: Skipped 1 previous similar message [ 6180.595541] Lustre: lustre-MDT0001: new disk, initializing [ 6180.696938] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6180.704936] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6183.738191] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 6190.543970] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6193.997972] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6198.050615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6198.215240] Lustre: lustre-OST0000: new disk, initializing [ 6198.218527] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6198.223614] Lustre: 209726:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6201.919719] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 6206.995064] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6207.005487] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6207.078834] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6208.018327] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6211.015136] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6211.149753] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6214.119128] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6214.233779] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6217.190928] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6217.191202] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6217.193422] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6217.193428] Lustre: Skipped 7 previous similar messages [ 6217.194597] LustreError: Skipped 1 previous similar message [ 6217.203767] Lustre: Skipped 5 previous similar messages [ 6221.498571] Lustre: server umount lustre-OST0000 complete [ 6230.305908] Lustre: server umount lustre-MDT0000 complete [ 6232.541479] LustreError: 207748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788335682 with bad export cookie 14113712623379673573 [ 6232.548287] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6232.550102] LustreError: 207748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6232.558676] LustreError: 207769:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6232.558687] LustreError: 207769:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 6232.739472] Lustre: server umount lustre-MDT0001 complete [ 6238.163333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6241.524090] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 6248.148470] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6251.461838] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 6256.180948] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6259.900487] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6264.697450] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6269.038915] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 6273.486406] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6276.373476] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6284.933309] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec [ 6288.179494] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6288.380931] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6293.987960] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6293.995623] LustreError: Skipped 2 previous similar messages [ 6294.000114] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6294.007887] Lustre: Skipped 2 previous similar messages [ 6304.223477] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6304.376162] Lustre: server umount lustre-OST0000 complete [ 6309.252380] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6312.112793] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6339.189418] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6339.413224] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6339.418891] Lustre: Skipped 5 previous similar messages [ 6344.306885] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 6349.438457] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6350.695764] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6350.710232] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6355.817182] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6355.828106] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6359.902486] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 9 sec [ 6366.181664] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6366.185176] Lustre: Skipped 9 previous similar messages [ 6368.698533] Lustre: server umount lustre-OST0000 complete [ 6377.600481] Lustre: server umount lustre-MDT0000 complete [ 6380.410152] LustreError: 211921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788335830 with bad export cookie 14113712623379674455 [ 6380.411503] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6380.420085] LustreError: 211921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6380.699915] Lustre: server umount lustre-MDT0001 complete [ 6387.572639] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 03:57:17 (1788335837) [ 6392.120632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6392.421071] LustreError: 217962:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6392.432032] LustreError: 217962:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6395.377576] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 6401.332668] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6404.781477] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 6409.416198] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6413.066453] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6417.902643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6422.279477] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 6427.501270] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6430.108344] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6438.643416] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec [ 6441.734620] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6441.893597] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6446.610169] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6446.732672] Lustre: lustre-OST0001: new disk, initializing [ 6446.735700] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6446.742100] Lustre: 221389:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6450.798515] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 6456.872357] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6456.884043] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6456.955120] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6457.611106] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6460.397939] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6460.588987] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6463.013945] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6463.196563] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6467.046440] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6467.046661] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6467.053117] Lustre: Skipped 5 previous similar messages [ 6467.065888] LustreError: Skipped 5 previous similar messages [ 6470.417169] Lustre: server umount lustre-OST0001 complete [ 6474.770153] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6474.915171] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6477.543883] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6477.680450] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6505.026096] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6510.506655] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 6510.570512] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6510.586342] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6510.592654] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6510.603446] Lustre: Skipped 1 previous similar message [ 6516.438835] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6517.407828] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6524.770195] Lustre: server umount lustre-OST0001 complete [ 6525.942384] LustreError: 219781:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6525.959179] LustreError: 219781:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 22 previous similar messages [ 6554.079130] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6554.244977] Lustre: server umount lustre-OST0000 complete [ 6563.670647] Lustre: server umount lustre-MDT0000 complete [ 6566.421580] LustreError: 221387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788336016 with bad export cookie 14113712623379675421 [ 6566.429947] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6566.438293] LustreError: 221387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6572.707870] Lustre: server umount lustre-MDT0001 complete [ 6590.286904] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 04:00:39 (1788336039) [ 6596.037824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6596.343167] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6596.376165] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6596.611654] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6596.614696] Lustre: Skipped 5 previous similar messages [ 6600.993761] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 6608.539771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6608.607702] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6608.629369] Lustre: 226626:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6613.215111] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 6619.997939] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6624.405267] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6630.067292] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6630.195863] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6635.361617] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 6641.487464] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6648.782536] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6648.838485] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6653.256305] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 6659.187782] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6660.490693] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6663.851471] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6664.004921] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6667.184660] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6667.333163] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6671.591144] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9a8e07a8d000.ost_server_uuid 50 [ 6672.793169] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9a8e07a8d000.ost_server_uuid in IDLE state after 0 sec [ 6673.554657] Lustre: Permanently deactivating lustre-OST0001 [ 6673.563751] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6676.333338] Lustre: Permanently reactivating lustre-OST0001 [ 6680.034926] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6680.042066] Lustre: Skipped 19 previous similar messages [ 6685.260070] Lustre: server umount lustre-OST0001 complete [ 6693.871703] Lustre: server umount lustre-OST0000 complete [ 6702.974637] Lustre: server umount lustre-MDT0000 complete [ 6705.255779] LustreError: 226619:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788336155 with bad export cookie 14113712623379676709 [ 6705.257773] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6705.267839] LustreError: 226619:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6705.488701] Lustre: server umount lustre-MDT0001 complete [ 6722.052569] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 04:02:51 (1788336171) [ 6727.065397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6727.247051] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6727.269482] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6727.278199] Lustre: Skipped 5 previous similar messages [ 6730.105265] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 6735.795963] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6735.914806] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6735.942462] Lustre: 233325:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6738.817630] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 6743.986361] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6747.870480] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6752.400509] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6752.555460] Lustre: Permanently deactivating lustre-OST0000 [ 6752.564310] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6753.098614] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6756.928459] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 6761.861643] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6766.434059] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6770.153823] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 6774.853491] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6774.915327] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6775.974814] Lustre: Permanently reactivating lustre-OST0000 [ 6775.985702] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6775.988471] Lustre: Skipped 2 previous similar messages [ 6785.063163] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6785.068888] Lustre: Skipped 13 previous similar messages [ 6785.072679] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6785.075578] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 32 seconds. I think it's dead, and I am evicting it. exp 000000005262b955, cur 1788336235 deadline 1788336233 last 1788336203 [ 6785.075711] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6785.107701] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6785.138882] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6789.092508] LustreError: lustre-OST0001-osc-MDT0001: operation ost_destroy to node 0@lo failed: rc = -107 [ 6789.103200] LustreError: Skipped 8 previous similar messages [ 6794.467716] Lustre: server umount lustre-OST0001 complete [ 6795.757663] LustreError: 236869:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6795.774685] LustreError: 236869:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [ 6803.333607] Lustre: server umount lustre-OST0000 complete [ 6820.319126] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6820.432633] Lustre: server umount lustre-MDT0000 complete [ 6822.797734] LustreError: 235203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788336272 with bad export cookie 14113712623379678354 [ 6822.801291] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6822.803613] LustreError: 235203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6823.007990] Lustre: server umount lustre-MDT0001 complete [ 6830.038643] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 6831.920234] Key type lgssc unregistered [ 6832.125680] LNet: 238199:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6832.130918] LNetError: 238199:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6833.189424] LNet: Removed LNI 192.168.203.160@tcp [ 6833.802252] Key type .llcrypt unregistered [ 6833.806249] Key type ._llcrypt unregistered [ 6848.152713] Key type ._llcrypt registered [ 6848.158827] Key type .llcrypt registered [ 6848.236427] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 6848.810788] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6848.856585] alg: No test for adler32 (adler32-zlib) [ 6849.799825] Lustre: Lustre: Build Version: 2.17.57_104_g6cc0f2c [ 6849.936841] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 6851.543177] Key type lgssc registered [ 6852.255423] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6858.162590] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6858.182143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6859.543178] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6862.091495] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 6867.004945] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6867.109296] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6867.135092] Lustre: 239628:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6867.143795] Lustre: Permanently deactivating lustre-MDT0001 [ 6867.153182] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6867.357698] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6867.525842] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6867.530425] LustreError: 239671:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 6867.538338] LustreError: 239671:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 6867.546616] LustreError: 239671:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 8, retries 1, failed: rc = -108 [ 6869.823613] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 6874.478132] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6878.419050] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6882.710550] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6882.905403] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6884.962113] LustreError: 241479:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6884.973232] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 6884.974134] LustreError: 241479:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6886.405548] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 6890.467603] LustreError: 241481:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6890.790370] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6892.517787] LustreError: 241479:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6892.538905] LustreError: 241479:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6895.123108] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6895.283558] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6899.282057] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 6900.719482] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 6900.722437] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 6903.708517] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6906.950265] Lustre: Permanently reactivating lustre-MDT0001 [ 6906.969459] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 6906.977494] Lustre: Skipped 2 previous similar messages [ 6909.704930] Lustre: lustre-MDT0001: Client da78af17-e152-49c4-aca5-3f5072be6620 (at 192.168.203.60@tcp) reconnecting [ 6914.208176] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6915.215503] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6918.596505] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6918.751971] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6921.148829] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6921.299079] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6922.852404] Lustre: Permanently deactivating lustre-MDT0001 [ 6922.867901] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6922.872814] Lustre: Skipped 2 previous similar messages [ 6930.482220] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6937.569713] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6937.571368] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6937.575067] Lustre: Skipped 2 previous similar messages [ 6937.583556] Lustre: Skipped 3 previous similar messages [ 6940.599086] Lustre: server umount lustre-MDT0000 complete [ 6942.693112] LustreError: 239641:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6942.704099] LustreError: 239641:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6942.828714] LustreError: 241500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788336392 with bad export cookie 2173126721075405911 [ 6942.831340] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6942.836265] LustreError: 241500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6943.010713] Lustre: server umount lustre-MDT0001 complete [ 6951.235129] Lustre: server umount lustre-OST0000 complete [ 6957.989500] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 04:06:47 (1788336407) [ 6964.594901] Lustre: server umount lustre-OST0001 complete [ 6968.652811] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_hostid [ 6974.383835] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 6994.978810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6995.123519] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6995.128966] Lustre: Skipped 2 previous similar messages [ 6995.141839] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6995.196475] Lustre: lustre-MDT0000: new disk, initializing [ 6995.252081] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6995.262494] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6997.926319] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 7004.815359] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7004.899792] Lustre: 247890:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7004.952666] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7004.954689] Lustre: Skipped 1 previous similar message [ 7005.004188] Lustre: lustre-MDT0001: new disk, initializing [ 7005.065567] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7005.093285] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7005.097937] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7007.577347] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 7013.512091] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7016.693560] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7020.375553] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7020.527181] Lustre: lustre-OST0000: new disk, initializing [ 7020.530898] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7020.535212] Lustre: 249857:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7024.096973] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 7028.250112] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7028.257098] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7028.314044] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7030.285974] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7031.810475] LustreError: 247898:0:(mdt_handler.c:3275:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7038.231036] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7038.364400] Lustre: lustre-OST0001: new disk, initializing [ 7038.369831] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7038.373901] Lustre: 250973:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7038.449627] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7038.455994] Lustre: Skipped 1 previous similar message [ 7041.831058] LustreError: 247898:0:(mdt_handler.c:3275:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7041.847535] LustreError: 250312:0:(mdt_handler.c:3275:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7042.405951] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 7046.694705] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7046.702592] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7046.773539] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7048.489513] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7049.481171] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7051.871182] LustreError: 250312:0:(mdt_handler.c:3275:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7055.565752] Lustre: lustre-OST0001: Not available for connect from 192.168.203.60@tcp (stopping) [ 7056.864695] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7056.872508] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7056.875522] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7056.891188] Lustre: Skipped 2 previous similar messages [ 7056.902504] Lustre: Skipped 1 previous similar message [ 7059.225382] Lustre: server umount lustre-OST0001 complete [ 7060.663672] LustreError: 249840:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7072.226991] LustreError: 249840:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7072.236482] LustreError: 249840:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 7074.272744] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7074.272885] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7074.278566] LustreError: Skipped 1 previous similar message [ 7074.280152] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7074.287621] Lustre: Skipped 1 previous similar message [ 7079.244396] Lustre: server umount lustre-OST0000 complete [ 7081.952288] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7081.955428] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7081.966024] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7081.968132] Lustre: Skipped 3 previous similar messages [ 7087.701319] Lustre: server umount lustre-MDT0000 complete [ 7089.773845] LustreError: 247883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788336539 with bad export cookie 2173126721075408501 [ 7089.780327] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7089.784241] LustreError: 247883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7096.152526] Lustre: server umount lustre-MDT0001 complete [ 7102.386633] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 7104.128541] Key type lgssc unregistered [ 7104.313203] LNet: 253003:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7104.318471] LNetError: 253003:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7104.335606] LNet: Removed LNI 192.168.203.160@tcp [ 7104.842847] Key type .llcrypt unregistered [ 7104.845181] Key type ._llcrypt unregistered [ 7126.093167] Key type ._llcrypt registered [ 7126.095174] Key type .llcrypt registered [ 7126.147717] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 7126.591977] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7126.650578] alg: No test for adler32 (adler32-zlib) [ 7127.565072] Lustre: Lustre: Build Version: 2.17.57_104_g6cc0f2c [ 7127.693488] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 7129.287121] Key type lgssc registered [ 7129.764545] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7133.130443] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7133.143577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7134.266087] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7134.279457] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7134.362518] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7136.250234] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 7140.476037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7140.533064] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7140.558312] Lustre: 255185:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7140.716916] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7142.831905] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 7146.369667] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7149.169858] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7152.216243] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7152.324749] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7152.433529] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7155.216603] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 7158.311943] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7161.373600] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7207.393518] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7207.398527] LustreError: Skipped 1 previous similar message [ 7207.398555] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7207.402223] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7207.410664] Lustre: Skipped 1 previous similar message [ 7207.417556] Lustre: Skipped 1 previous similar message [ 7209.780265] Lustre: server umount lustre-OST0000 complete [ 7211.837554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7217.586819] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7217.805721] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7220.670349] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 7224.476478] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7228.070916] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7228.079607] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7228.081365] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7228.091030] Lustre: Skipped 1 previous similar message [ 7259.107618] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7259.110289] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7259.112178] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7259.117716] Lustre: Skipped 1 previous similar message [ 7264.232090] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7264.236021] Lustre: Skipped 2 previous similar messages [ 7264.973673] Lustre: server umount lustre-OST0000 complete [ 7268.832233] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7268.838401] LustreError: Skipped 1 previous similar message [ 7268.840191] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7268.847479] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7274.464290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7274.469180] Lustre: Skipped 3 previous similar messages [ 7281.119158] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7281.223817] Lustre: server umount lustre-MDT0000 complete [ 7282.943682] LustreError: 255177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788336732 with bad export cookie 5777686877903384948 [ 7282.948617] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7282.951105] LustreError: 255177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7283.099625] Lustre: server umount lustre-MDT0001 complete [ 7288.047153] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 7289.419505] Key type lgssc unregistered [ 7289.578429] LNet: 260598:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7289.582309] LNetError: 260598:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7289.591580] LNet: Removed LNI 192.168.203.160@tcp [ 7290.024912] Key type .llcrypt unregistered [ 7290.027859] Key type ._llcrypt unregistered [ 7303.486284] Key type ._llcrypt registered [ 7303.488303] Key type .llcrypt registered [ 7303.546429] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 7303.998960] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7304.042551] alg: No test for adler32 (adler32-zlib) [ 7304.978900] Lustre: Lustre: Build Version: 2.17.57_104_g6cc0f2c [ 7305.123507] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 7306.743314] Key type lgssc registered [ 7307.287295] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7311.130274] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7311.139760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7312.356087] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7314.563852] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 7319.099151] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7319.280144] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7321.407753] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 7325.321308] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7328.183456] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7331.917765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7332.154047] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7335.541613] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 7337.263662] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7339.609653] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7357.921362] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7357.925852] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7357.932637] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7359.970705] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7359.972853] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7359.982129] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7362.615573] Lustre: server umount lustre-OST0000 complete [ 7365.088718] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7365.090324] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7365.095229] Lustre: Skipped 1 previous similar message [ 7365.100846] Lustre: Skipped 2 previous similar messages [ 7370.208696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7370.212595] Lustre: Skipped 1 previous similar message [ 7370.617417] Lustre: server umount lustre-MDT0000 complete [ 7372.424143] LustreError: 263829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788336822 with bad export cookie 16274545398738448842 [ 7372.424974] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7372.433539] LustreError: 263829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7372.595588] Lustre: server umount lustre-MDT0001 complete [ 7377.855048] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 7379.293981] Key type lgssc unregistered [ 7379.461101] LNet: 266286:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7379.465453] LNetError: 266286:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7380.518877] LNet: Removed LNI 192.168.203.160@tcp [ 7380.928685] Key type .llcrypt unregistered [ 7380.930225] Key type ._llcrypt unregistered [ 7390.079086] Key type ._llcrypt registered [ 7390.080691] Key type .llcrypt registered [ 7390.135260] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 7390.658555] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7390.668705] alg: No test for adler32 (adler32-zlib) [ 7391.622855] Lustre: Lustre: Build Version: 2.17.57_104_g6cc0f2c [ 7391.754494] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 7393.359158] Key type lgssc registered [ 7393.980804] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7398.154347] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7398.166774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7399.380880] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7401.593026] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 7406.042733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7406.272445] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7408.483197] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 7411.986856] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7414.882810] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7418.321434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7418.492043] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7421.259233] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 7424.395334] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7424.619971] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7440.352974] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7440.359192] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7440.367993] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7442.403744] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7442.404088] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7442.410798] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7445.234662] Lustre: server umount lustre-OST0000 complete [ 7447.520606] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7447.521119] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7447.522943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7447.522950] Lustre: Skipped 1 previous similar message [ 7447.529214] Lustre: Skipped 1 previous similar message [ 7452.642797] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7452.645995] Lustre: Skipped 2 previous similar messages [ 7453.284620] Lustre: server umount lustre-MDT0000 complete [ 7454.932506] LustreError: 267462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788336904 with bad export cookie 9704473777894995323 [ 7454.938070] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7454.942670] LustreError: 267462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7455.111401] Lustre: server umount lustre-MDT0001 complete [ 7459.774675] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 7461.154639] Key type lgssc unregistered [ 7461.322567] LNet: 271176:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7461.326960] LNetError: 271176:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7461.344563] LNet: Removed LNI 192.168.203.160@tcp [ 7461.744433] Key type .llcrypt unregistered [ 7461.747960] Key type ._llcrypt unregistered [ 7473.443672] Key type ._llcrypt registered [ 7473.445068] Key type .llcrypt registered [ 7473.501754] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 7473.830706] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7473.842655] alg: No test for adler32 (adler32-zlib) [ 7474.729869] Lustre: Lustre: Build Version: 2.17.57_104_g6cc0f2c [ 7474.838641] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 7476.423223] Key type lgssc registered [ 7476.918729] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7480.403464] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7480.416116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7481.649948] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7483.456713] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 7487.936852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7488.169445] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7490.212205] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 7494.039386] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7496.842937] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7500.158842] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7500.330229] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7503.093422] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 7506.320297] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7506.475960] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7526.881250] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7526.884916] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7526.891402] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7528.929191] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7528.930736] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7528.933306] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7528.943581] Lustre: Skipped 1 previous similar message [ 7532.645431] Lustre: server umount lustre-OST0000 complete [ 7534.560358] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7534.567359] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7534.573480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7539.171236] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7539.173354] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7539.183317] Lustre: Skipped 1 previous similar message [ 7540.643168] Lustre: server umount lustre-MDT0000 complete [ 7542.333725] LustreError: 272546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788336992 with bad export cookie 8106066813184978110 [ 7542.336672] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7542.338285] LustreError: 272546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7542.505379] Lustre: server umount lustre-MDT0001 complete [ 7547.401719] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 7548.776448] Key type lgssc unregistered [ 7548.954760] LNet: 276887:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7548.958546] LNetError: 276887:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7548.968734] LNet: Removed LNI 192.168.203.160@tcp [ 7549.421778] Key type .llcrypt unregistered [ 7549.424243] Key type ._llcrypt unregistered [ 7557.966236] Key type ._llcrypt registered [ 7557.967437] Key type .llcrypt registered [ 7558.015759] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 7558.470052] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7558.506964] alg: No test for adler32 (adler32-zlib) [ 7559.410211] Lustre: Lustre: Build Version: 2.17.57_104_g6cc0f2c [ 7559.533324] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 7561.127323] Key type lgssc registered [ 7561.622443] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7565.087406] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7565.096788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7566.310062] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7568.051361] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 7572.089757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7572.276930] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7574.141525] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 7577.569995] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7579.983508] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7583.130349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7583.290804] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7585.900893] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 7589.119753] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7591.464941] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 7606.752397] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7606.753628] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7606.756046] LustreError: Skipped 1 previous similar message [ 7606.762739] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7607.778336] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7607.780837] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7607.781390] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7610.085208] Lustre: server umount lustre-OST0000 complete [ 7612.037457] Lustre: server umount lustre-MDT0000 complete [ 7612.898196] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7612.901858] LustreError: 278099:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7612.903994] Lustre: Skipped 1 previous similar message [ 7612.918037] LustreError: 278099:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7613.959244] LustreError: 279961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788337063 with bad export cookie 17963979002373812708 [ 7613.960160] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7613.964369] LustreError: 279961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7620.173186] Lustre: server umount lustre-MDT0001 complete [ 7625.233070] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 7626.535714] Key type lgssc unregistered [ 7626.690638] LNet: 281813:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7626.695893] LNetError: 281813:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7626.706582] LNet: Removed LNI 192.168.203.160@tcp [ 7627.111705] Key type .llcrypt unregistered [ 7627.114506] Key type ._llcrypt unregistered [ 7632.425655] Key type ._llcrypt registered [ 7632.427542] Key type .llcrypt registered [ 7632.483625] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 7639.917191] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_hostid [ 7647.764856] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 7648.257468] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7648.265236] alg: No test for adler32 (adler32-zlib) [ 7649.169626] Lustre: Lustre: Build Version: 2.17.57_104_g6cc0f2c [ 7649.310356] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 7650.912627] Key type lgssc registered [ 7651.458937] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7666.805605] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7666.816120] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7667.942765] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7667.957918] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7668.016811] Lustre: lustre-MDT0000: new disk, initializing [ 7668.064639] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7668.075135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7669.952537] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 7675.294933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7675.352807] Lustre: 285334:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7675.407796] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7675.411299] Lustre: Skipped 1 previous similar message [ 7675.467224] Lustre: lustre-MDT0001: new disk, initializing [ 7675.520301] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7675.543119] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7675.549832] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7677.557728] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 7682.525094] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7685.028912] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7688.021206] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7688.149141] Lustre: lustre-OST0000: new disk, initializing [ 7688.151573] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7688.155830] Lustre: 287300:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7688.201541] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7689.607269] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7689.612593] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7689.685079] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7690.851148] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 7695.617548] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7697.518054] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7697.608076] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7699.551125] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7699.640897] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7705.059041] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7705.059911] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7705.064032] LustreError: Skipped 1 previous similar message [ 7705.070308] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7705.074954] Lustre: Skipped 1 previous similar message [ 7706.595993] Lustre: server umount lustre-OST0000 complete [ 7710.178629] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7710.179669] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7710.184547] Lustre: Skipped 2 previous similar messages [ 7714.557632] Lustre: server umount lustre-MDT0000 complete [ 7715.296465] LustreError: 285346:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7715.305039] LustreError: 285346:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7716.188496] LustreError: 285328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788337166 with bad export cookie 16036908738999283480 [ 7716.191966] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7716.195337] LustreError: 285328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7716.323589] Lustre: server umount lustre-MDT0001 complete [ 7719.968973] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 04:19:30 (1788337170) [ 7722.717383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7722.934670] LustreError: 289708:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7722.966537] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7724.785729] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 7728.098043] LustreError: 289709:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7728.552808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7728.744435] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7730.795654] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 7734.086907] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7736.822709] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7740.220300] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7740.415163] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7743.344510] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 7747.343101] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7757.792163] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7757.797155] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7757.804100] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7757.807048] Lustre: Skipped 1 previous similar message [ 7758.817647] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7762.364968] Lustre: server umount lustre-OST0000 complete [ 7764.449753] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7764.453790] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7764.460490] Lustre: Skipped 1 previous similar message [ 7764.464977] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7764.468031] Lustre: Skipped 2 previous similar messages [ 7766.462977] Lustre: server umount lustre-MDT0000 complete [ 7768.292522] LustreError: 289689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788337218 with bad export cookie 16036908738999284355 [ 7768.296394] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7768.297459] LustreError: 289689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7768.468716] Lustre: server umount lustre-MDT0001 complete [ 7773.669072] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 7774.943829] Key type lgssc unregistered [ 7775.109320] LNet: 293241:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7775.112114] LNetError: 293241:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7775.121339] LNet: Removed LNI 192.168.203.160@tcp [ 7775.536361] Key type .llcrypt unregistered [ 7775.537883] Key type ._llcrypt unregistered [ 7787.762190] Key type ._llcrypt registered [ 7787.763366] Key type .llcrypt registered [ 7787.822821] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_hostid [ 7795.106917] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 7795.635105] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7795.656099] alg: No test for adler32 (adler32-zlib) [ 7796.546707] Lustre: Lustre: Build Version: 2.17.57_104_g6cc0f2c [ 7796.676379] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 7798.279218] Key type lgssc registered [ 7798.764152] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7818.640024] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7818.648733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7819.737344] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7819.748643] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7819.792629] Lustre: lustre-MDT0000: new disk, initializing [ 7819.829388] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7819.841676] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7821.481796] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 7826.330343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7826.384163] Lustre: 297033:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7826.422664] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7826.426081] Lustre: Skipped 1 previous similar message [ 7826.467975] Lustre: lustre-MDT0001: new disk, initializing [ 7826.512749] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7826.532499] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7826.537679] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7828.233898] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 7832.469155] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7834.653592] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7837.381498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7837.497911] Lustre: lustre-OST2710: new disk, initializing [ 7837.501156] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 7837.505730] Lustre: 299001:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7837.543431] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 7839.859105] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 7844.438293] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7846.413796] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 7846.419475] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 7846.474377] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 7847.937086] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7848.019282] Lustre: lustre-OST03e8: new disk, initializing [ 7848.021549] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 7848.023911] Lustre: 300066:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7848.067280] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 7850.855791] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 7855.144861] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 7855.151880] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 7855.250192] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 7856.270258] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7863.451820] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 7863.599359] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7867.785068] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 7868.009179] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7871.741654] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 7871.899876] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7875.784871] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 7875.981444] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7881.206132] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7881.209170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7881.214955] Lustre: Skipped 1 previous similar message [ 7881.222235] Lustre: Skipped 2 previous similar messages [ 7886.157791] Lustre: server umount lustre-MDT0000 complete [ 7889.113449] LustreError: 297027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788337339 with bad export cookie 17786385415098184350 [ 7889.115259] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7889.123702] LustreError: 297027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7889.416621] Lustre: server umount lustre-MDT0001 complete [ 7902.653838] Lustre: server umount lustre-OST2710 complete [ 7915.952363] Lustre: server umount lustre-OST03e8 complete [ 7929.708127] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_hostid [ 7936.455183] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 7956.892985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7957.027219] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7957.045493] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7957.097722] Lustre: lustre-MDT0000: new disk, initializing [ 7957.139758] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7957.155471] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7959.361437] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 7965.304767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7965.365667] Lustre: 305309:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7965.403942] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7965.406922] Lustre: Skipped 1 previous similar message [ 7965.477425] Lustre: lustre-MDT0001: new disk, initializing [ 7965.549377] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7965.573935] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7965.582037] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7967.989104] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 7974.013527] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7977.099654] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7981.122069] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7981.253975] Lustre: 307274:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7984.545479] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 7989.783235] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7989.789856] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7989.863816] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7990.603042] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7992.850683] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7992.996899] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7995.153568] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7995.257744] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7999.977515] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7999.982203] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7999.984874] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7999.984885] Lustre: Skipped 1 previous similar message [ 7999.985961] LustreError: Skipped 1 previous similar message [ 8000.003051] Lustre: Skipped 3 previous similar messages [ 8002.542842] Lustre: server umount lustre-OST0000 complete [ 8005.091072] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8005.101366] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8005.104494] Lustre: Skipped 2 previous similar messages [ 8010.211147] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8010.217125] Lustre: Skipped 1 previous similar message [ 8010.817885] Lustre: server umount lustre-MDT0000 complete [ 8013.088119] LustreError: 306112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788337463 with bad export cookie 17786385415098186023 [ 8013.089099] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8013.098840] LustreError: 306112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8013.338091] Lustre: server umount lustre-MDT0001 complete [ 8018.368339] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 04:24:28 (1788337468) [ 8019.334710] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8020.305240] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 04:24:30 (1788337470) [ 8022.199658] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 8037.711139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8037.834973] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8037.846545] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8037.926688] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8037.929012] Lustre: Skipped 1 previous similar message [ 8040.523335] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 8045.389756] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8045.456227] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8045.470887] Lustre: 311125:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8045.511508] LustreError: 311138:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8045.522607] LustreError: 311138:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 8047.873219] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 8065.573024] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8065.693200] Lustre: Denying initial registration attempt from nid 192.168.203.160@tcp, specified as failover [ 8065.697775] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 8065.701884] LustreError: 312579:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -99 [ 8065.705294] LustreError: 312579:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 8065.707390] LustreError: 312579:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 8065.737556] Lustre: server umount lustre-OST0000 complete [ 8065.740120] LustreError: 312579:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 8071.137290] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8071.138337] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8071.140594] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8071.154216] Lustre: Skipped 2 previous similar messages [ 8076.258197] LustreError: 311135:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8076.271690] LustreError: 311135:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8077.562101] Lustre: server umount lustre-MDT0000 complete [ 8079.486872] LustreError: 311120:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788337529 with bad export cookie 17786385415098186905 [ 8079.490203] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8079.494435] LustreError: 311120:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8079.641475] Lustre: server umount lustre-MDT0001 complete [ 8084.911687] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 8086.242405] Key type lgssc unregistered [ 8086.398934] LNet: 313699:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8086.405183] LNetError: 313699:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8086.416383] LNet: Removed LNI 192.168.203.160@tcp [ 8086.822435] Key type .llcrypt unregistered [ 8086.824414] Key type ._llcrypt unregistered [ 8091.624224] Key type ._llcrypt registered [ 8091.625221] Key type .llcrypt registered [ 8091.665777] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 8092.186516] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8092.206372] alg: No test for adler32 (adler32-zlib) [ 8093.104757] Lustre: Lustre: Build Version: 2.17.57_104_g6cc0f2c [ 8093.210883] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 8094.823179] Key type lgssc registered [ 8095.333259] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8124.064713] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 8127.814568] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8127.824657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8128.943451] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8128.953823] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8129.017284] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8130.704568] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 8134.459469] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8134.511695] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8134.532252] Lustre: 316433:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8134.667904] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8136.598807] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 8139.678805] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8142.003379] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8158.317403] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8158.428881] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8158.529589] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8161.746798] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 8165.702752] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8166.851708] Lustre: server umount lustre-OST0000 complete [ 8170.465861] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8170.469347] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8170.483319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8173.536619] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8173.537141] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8174.800327] Lustre: server umount lustre-MDT0000 complete [ 8176.630459] LustreError: 316427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788337626 with bad export cookie 5880632996126969131 [ 8176.632439] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8176.637151] LustreError: 316427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8176.835690] Lustre: server umount lustre-MDT0001 complete [ 8181.818389] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 8183.152900] Key type lgssc unregistered [ 8183.297490] LNet: 320001:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8183.300920] LNetError: 320001:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8183.309893] LNet: Removed LNI 192.168.203.160@tcp [ 8183.729158] Key type .llcrypt unregistered [ 8183.732613] Key type ._llcrypt unregistered [ 8195.634553] Key type ._llcrypt registered [ 8195.636282] Key type .llcrypt registered [ 8195.688273] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 8196.258064] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8196.275171] alg: No test for adler32 (adler32-zlib) [ 8197.151870] Lustre: Lustre: Build Version: 2.17.57_104_g6cc0f2c [ 8197.292927] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 8198.895143] Key type lgssc registered [ 8199.387472] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8202.633829] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8202.642805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8203.839815] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8205.720360] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 8209.846102] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8210.051574] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8211.924300] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 8215.260281] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8217.615299] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8222.563046] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8222.725981] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8225.380138] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 8228.790894] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8240.098294] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8240.104351] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8240.108138] Lustre: Skipped 2 previous similar messages [ 8244.210124] Lustre: server umount lustre-MDT0000 complete [ 8245.216775] LustreError: 321385:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8245.224373] LustreError: 321385:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 8246.038670] LustreError: 321372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788337695 with bad export cookie 18307996370485237158 [ 8246.042649] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8246.044595] LustreError: 321372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8250.335799] LustreError: 323049:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8250.336324] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8250.344600] LustreError: 323049:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8250.350793] Lustre: Skipped 2 previous similar messages [ 8250.357166] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8250.363428] Lustre: Skipped 1 previous similar message [ 8251.360221] LustreError: 321386:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8251.360232] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8251.360238] Lustre: Skipped 1 previous similar message [ 8251.369095] LustreError: 321386:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8252.233396] Lustre: server umount lustre-MDT0001 complete [ 8255.254847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8259.115315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8271.841216] LustreError: 320700:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000ba850172 x1875207936567808/t0(0) o250->MGC192.168.203.160@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8271.995780] LustreError: 323219:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8272.045688] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8272.076452] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 8273.818910] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 8276.130377] LustreError: 324980:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8276.140118] LustreError: 324980:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8277.851433] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8278.062743] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8279.865663] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 8281.126486] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8283.326088] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8286.136192] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8288.774757] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8291.619358] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8296.418804] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8296.421224] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8296.428178] Lustre: Skipped 1 previous similar message [ 8301.538077] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8301.541213] Lustre: Skipped 2 previous similar messages [ 8308.192372] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8308.267315] Lustre: server umount lustre-OST0000 complete [ 8311.777223] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8311.783853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8311.787121] Lustre: Skipped 3 previous similar messages [ 8316.064532] Lustre: server umount lustre-MDT0000 complete [ 8316.898356] LustreError: 324984:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8316.906927] LustreError: 324984:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8317.702721] LustreError: 321371:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788337767 with bad export cookie 18307996370485244312 [ 8317.706114] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8317.709108] LustreError: 321371:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8317.862181] Lustre: server umount lustre-MDT0001 complete [ 8322.515239] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 8323.803268] Key type lgssc unregistered [ 8323.960675] LNet: 327649:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8323.965958] LNetError: 327649:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8323.977387] LNet: Removed LNI 192.168.203.160@tcp [ 8324.367567] Key type .llcrypt unregistered [ 8324.369201] Key type ._llcrypt unregistered [ 8335.610501] Key type ._llcrypt registered [ 8335.612056] Key type .llcrypt registered [ 8335.660314] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 8336.054235] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8336.121323] alg: No test for adler32 (adler32-zlib) [ 8337.013779] Lustre: Lustre: Build Version: 2.17.57_104_g6cc0f2c [ 8337.141696] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 8338.727143] Key type lgssc registered [ 8339.181798] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8342.641872] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8342.652090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8343.877295] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8345.642053] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 8349.493712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8349.662797] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8351.324262] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 8354.436202] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8356.744416] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8358.880874] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8358.882252] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8358.886619] Lustre: Skipped 1 previous similar message [ 8359.906783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8359.910447] Lustre: Skipped 2 previous similar messages [ 8365.024586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8365.027725] Lustre: Skipped 1 previous similar message [ 8370.144792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8370.149259] Lustre: Skipped 1 previous similar message [ 8372.192807] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8372.262380] Lustre: server umount lustre-MDT0000 complete [ 8373.846694] LustreError: 329021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788337823 with bad export cookie 8799274510238872285 [ 8373.850299] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8373.853990] LustreError: 329021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8374.001375] Lustre: server umount lustre-MDT0001 complete [ 8380.799799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8380.948646] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8380.964106] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8381.059877] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8382.775471] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 8386.634915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8386.701320] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8386.716537] Lustre: 331624:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8386.840770] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8388.637361] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 8391.987680] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8394.459060] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8399.913142] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8400.046068] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 8400.052435] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 8400.057429] LustreError: 333511:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -2 [ 8400.061821] LustreError: 333511:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 8400.064471] LustreError: 333511:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 8400.097589] Lustre: server umount lustre-OST0000 complete [ 8400.099203] LustreError: 333511:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 8405.742391] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8405.846071] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8405.935711] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8408.463917] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 8411.377656] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8412.710897] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 8414.318447] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8414.354037] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 8414.388565] Lustre: lustre-OST0001: new disk, initializing [ 8414.390657] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8414.393437] Lustre: 335050:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8414.429115] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8416.828088] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 8419.336886] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8419.341509] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8419.388874] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8419.737706] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8424.417234] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8424.417909] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8424.420608] LustreError: Skipped 1 previous similar message [ 8424.421164] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8424.425748] Lustre: Skipped 1 previous similar message [ 8424.432241] Lustre: Skipped 1 previous similar message [ 8427.010742] Lustre: server umount lustre-OST0001 complete [ 8429.538335] LustreError: 334794:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8429.538771] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8429.545591] LustreError: 334794:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8434.522198] Lustre: server umount lustre-OST0000 complete [ 8438.240100] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8438.244117] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8438.249167] Lustre: Skipped 1 previous similar message [ 8438.251209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8438.253428] Lustre: Skipped 2 previous similar messages [ 8442.335421] Lustre: server umount lustre-MDT0000 complete [ 8443.727413] LustreError: 331617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788337893 with bad export cookie 8799274510238872824 [ 8443.729966] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8443.732306] LustreError: 331617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8443.852319] Lustre: server umount lustre-MDT0001 complete [ 8447.726589] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 8448.881560] Key type lgssc unregistered [ 8449.021966] LNet: 337005:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8449.026362] LNetError: 337005:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8449.035491] LNet: Removed LNI 192.168.203.160@tcp [ 8449.370133] Key type .llcrypt unregistered [ 8449.371432] Key type ._llcrypt unregistered [ 8482.736160] Key type ._llcrypt registered [ 8482.737477] Key type .llcrypt registered [ 8482.787032] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_hostid [ 8489.306315] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 8489.763153] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8489.783481] alg: No test for adler32 (adler32-zlib) [ 8490.645246] Lustre: Lustre: Build Version: 2.17.57_104_g6cc0f2c [ 8490.746810] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 8492.335217] Key type lgssc registered [ 8492.747391] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8506.197200] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8506.210028] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8507.315982] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8507.337808] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8507.384430] Lustre: lustre-MDT0000: new disk, initializing [ 8507.421493] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8507.429958] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8509.013320] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 8513.495481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8513.545833] Lustre: 341900:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8513.574299] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8513.576466] Lustre: Skipped 1 previous similar message [ 8513.613094] Lustre: lustre-MDT0001: new disk, initializing [ 8513.653433] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8513.668814] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8513.675190] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8515.146039] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 8519.204607] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8521.249705] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8523.741949] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8523.851271] Lustre: lustre-OST0000: new disk, initializing [ 8523.853449] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8523.857253] Lustre: 343870:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8523.889282] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8526.043582] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 8529.407951] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8529.412190] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8529.449786] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8530.094918] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8531.663204] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8531.745611] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8533.277315] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8533.364226] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8534.498240] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8534.498612] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8534.502519] LustreError: Skipped 1 previous similar message [ 8534.508938] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8534.513188] Lustre: Skipped 1 previous similar message [ 8536.147241] Lustre: server umount lustre-OST0000 complete [ 8539.616784] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8539.617087] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8539.620335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8539.625058] Lustre: Skipped 2 previous similar messages [ 8543.648455] Lustre: server umount lustre-MDT0000 complete [ 8545.103357] LustreError: 341895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788337995 with bad export cookie 10511533433118739997 [ 8545.105818] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8545.107056] LustreError: 341895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8545.227697] Lustre: server umount lustre-MDT0001 complete [ 8548.260687] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 04:33:18 (1788337998) [ 8550.890274] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 04:33:21 (1788338001) [ 8554.089020] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8554.252352] LustreError: 346622:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8554.259265] LustreError: 346622:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8554.278153] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8555.775964] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 8558.991527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8559.153613] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8560.726575] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 8563.500907] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8565.493621] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8567.877174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8567.998648] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8570.216366] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 8573.079469] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8588.670749] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 8589.391559] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 8590.055496] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 8590.745204] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 8591.509760] Lustre: Failing over lustre-MDT0000 [ 8591.729047] Lustre: server umount lustre-MDT0000 complete [ 8594.913657] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8594.916750] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8594.922271] LustreError: 349118:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8595.936435] LustreError: 346641:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8595.943389] LustreError: 346641:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8600.240894] LustreError: 347965:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8605.108849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8605.164425] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8605.284604] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8605.317235] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 8605.361133] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8606.919602] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 8610.791680] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8610.813503] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 8610.831072] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 8613.061716] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8613.669671] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8615.752873] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 8616.563164] Lustre: Failing over lustre-MDT0000 [ 8616.781876] Lustre: server umount lustre-MDT0000 complete [ 8620.728453] LustreError: 347965:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8620.735748] LustreError: 347965:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8621.026428] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8621.031637] Lustre: Skipped 4 previous similar messages [ 8621.034370] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8630.284729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8630.351648] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8630.497700] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 8630.960503] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8632.140643] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 8635.886208] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8635.889046] Lustre: Skipped 2 previous similar messages [ 8635.903812] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 8635.922993] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 8637.762642] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8638.312220] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8640.992293] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8640.994950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8640.998576] Lustre: Skipped 2 previous similar messages [ 8640.998947] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8641.001654] Lustre: Skipped 3 previous similar messages [ 8646.115031] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8646.117700] Lustre: Skipped 2 previous similar messages [ 8646.812869] Lustre: server umount lustre-MDT0000 complete [ 8648.232650] LustreError: 346604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788338098 with bad export cookie 10511533433118742678 [ 8648.236362] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8648.238561] LustreError: 346604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8648.377782] Lustre: server umount lustre-MDT0001 complete [ 8660.134179] Lustre: server umount lustre-OST0000 complete [ 8665.931803] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8666.108611] LustreError: 352974:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8666.116618] LustreError: 352974:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 8666.133524] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8666.136361] Lustre: Skipped 1 previous similar message [ 8667.685210] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 8670.931768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8672.727664] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 8675.595852] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8677.850261] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8680.551428] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8682.977221] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 8685.723227] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8692.007419] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 8702.436662] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8702.439430] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8702.439957] LustreError: Skipped 1 previous similar message [ 8702.440879] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8702.447271] Lustre: Skipped 1 previous similar message [ 8712.159129] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8712.238829] Lustre: server umount lustre-OST0000 complete [ 8717.281339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8717.290427] Lustre: Skipped 3 previous similar messages [ 8720.122060] Lustre: server umount lustre-MDT0000 complete [ 8722.678418] LustreError: 353709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788338172 with bad export cookie 10511533433118743427 [ 8722.679255] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8722.684129] LustreError: 353709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8723.037763] Lustre: server umount lustre-MDT0001 complete [ 8733.197244] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 8736.533773] Key type lgssc unregistered [ 8736.751886] LNet: 356525:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8736.757304] LNetError: 356525:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8736.768369] LNet: Removed LNI 192.168.203.160@tcp [ 8737.322114] Key type .llcrypt unregistered [ 8737.324080] Key type ._llcrypt unregistered [ 8765.211848] Key type ._llcrypt registered [ 8765.214926] Key type .llcrypt registered [ 8765.361359] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 8767.583669] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8767.764728] alg: No test for adler32 (adler32-zlib) [ 8769.035346] Lustre: Lustre: Build Version: 2.17.57_104_g6cc0f2c [ 8769.429817] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 8771.175885] Key type lgssc registered [ 8772.669655] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8782.355410] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8782.389383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8784.150591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8789.046835] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 8794.605796] LustreError: 358038:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8794.636094] LustreError: 358038:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8797.572752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8797.921528] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8801.955339] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 8807.220847] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8810.942934] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8814.992473] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8815.177302] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8818.621229] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 8823.010138] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8829.547834] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 8839.138037] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8839.142741] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8839.154649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8840.160524] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8840.163321] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8841.647686] Lustre: server umount lustre-MDT0000 complete [ 8843.865708] LustreError: 358018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788338293 with bad export cookie 4086818119748037808 [ 8843.874179] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8843.878556] LustreError: 358018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8844.072620] Lustre: server umount lustre-MDT0001 complete [ 8856.867461] Lustre: server umount lustre-OST0000 complete [ 8865.610588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8865.930625] LustreError: 361768:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8865.980427] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8868.641683] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 8871.392252] LustreError: 361769:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8873.890128] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8874.126114] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8876.457257] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 8880.666704] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8883.938820] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8887.895024] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8888.081259] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8890.159847] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 8891.674706] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 8895.801882] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8905.184681] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8905.191512] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8905.204352] Lustre: Skipped 1 previous similar message [ 8905.215192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8905.220643] Lustre: Skipped 2 previous similar messages [ 8909.795655] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8909.796153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8909.806370] Lustre: Skipped 1 previous similar message [ 8909.817990] Lustre: Skipped 2 previous similar messages [ 8911.197153] Lustre: server umount lustre-MDT0000 complete [ 8913.721874] LustreError: 361751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788338363 with bad export cookie 4086818119748038921 [ 8913.732477] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8913.732822] LustreError: 361751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 8913.941281] Lustre: server umount lustre-MDT0001 complete [ 8926.857984] Lustre: server umount lustre-OST0000 complete [ 8937.813308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8938.131836] LustreError: 365673:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8938.195411] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8941.860419] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 8943.584074] LustreError: 365674:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8948.489239] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8951.980417] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 8957.732194] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8962.189160] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8967.272973] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8971.969403] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 8977.531793] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8985.016245] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 8994.279753] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8994.284443] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8994.306650] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8998.039628] Lustre: server umount lustre-OST0000 complete [ 9005.027277] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9005.035869] LustreError: Skipped 1 previous similar message [ 9005.044149] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9005.055418] Lustre: Skipped 1 previous similar message [ 9005.061289] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9005.065829] Lustre: Skipped 2 previous similar messages [ 9007.423454] Lustre: server umount lustre-MDT0000 complete [ 9010.656263] LustreError: 365674:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9010.668843] LustreError: 365674:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9010.700069] LustreError: 365653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788338460 with bad export cookie 4086818119748039943 [ 9010.702421] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9010.710942] LustreError: 365653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9010.963559] Lustre: server umount lustre-MDT0001 complete [ 9018.964757] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 9021.247914] Key type lgssc unregistered [ 9021.473394] LNet: 369223:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9021.480489] LNetError: 369223:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9021.492893] LNet: Removed LNI 192.168.203.160@tcp [ 9022.108901] Key type .llcrypt unregistered [ 9022.110681] Key type ._llcrypt unregistered [ 9039.971534] Key type ._llcrypt registered [ 9039.973833] Key type .llcrypt registered [ 9040.069579] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 9040.605983] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9040.638680] alg: No test for adler32 (adler32-zlib) [ 9041.681269] Lustre: Lustre: Build Version: 2.17.57_104_g6cc0f2c [ 9041.854173] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 9043.527168] Key type lgssc registered [ 9044.383338] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9050.560134] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9050.588524] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9050.597663] LustreError: 370637:0:(osd_handler.c:8549:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 9050.634650] LustreError: 370637:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 9050.641874] LustreError: 370637:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 9050.650971] LustreError: 370637:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 9050.662522] LustreError: 370637:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 9056.350759] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9056.358809] LustreError: 371033:0:(osd_handler.c:8549:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 9056.374382] LustreError: 371033:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 9056.379817] LustreError: 371033:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 9056.387504] LustreError: 371033:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 9056.396932] LustreError: 371033:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 9070.757672] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 9072.847629] Key type lgssc unregistered [ 9073.082101] LNet: 372036:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9073.087934] LNetError: 372036:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9073.110311] LNet: Removed LNI 192.168.203.160@tcp [ 9073.673186] Key type .llcrypt unregistered [ 9073.675082] Key type ._llcrypt unregistered [ 9086.083644] Key type ._llcrypt registered [ 9086.085225] Key type .llcrypt registered [ 9086.152733] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_hostid [ 9096.967049] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 9097.645437] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9097.706232] alg: No test for adler32 (adler32-zlib) [ 9098.674721] Lustre: Lustre: Build Version: 2.17.57_104_g6cc0f2c [ 9098.884743] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 9100.537546] Key type lgssc registered [ 9101.340466] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9132.079646] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9132.125603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9133.409711] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9133.443968] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9133.513902] Lustre: lustre-MDT0000: new disk, initializing [ 9133.573515] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9133.599580] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9140.774091] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 9156.925641] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9157.372931] Lustre: 375164:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9157.514260] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9157.521661] Lustre: Skipped 1 previous similar message [ 9157.678614] Lustre: lustre-MDT0001: new disk, initializing [ 9157.949615] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9158.019366] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9158.039163] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9165.406950] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 9177.387698] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9184.874729] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9194.232495] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9194.786859] Lustre: lustre-OST0000: new disk, initializing [ 9194.790424] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9194.815792] Lustre: 377136:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9195.012794] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9201.882447] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 9203.276996] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9203.297362] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9203.470522] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9212.762414] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9217.931515] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9218.188603] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9224.246327] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9224.572661] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9229.283249] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9229.288920] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9229.298708] LustreError: Skipped 1 previous similar message [ 9229.306837] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9229.320610] Lustre: Skipped 1 previous similar message [ 9233.322513] Lustre: server umount lustre-OST0000 complete [ 9239.520899] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9239.523279] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9239.538817] Lustre: Skipped 2 previous similar messages [ 9243.435500] Lustre: server umount lustre-MDT0000 complete [ 9247.086579] LustreError: 375156:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788338697 with bad export cookie 14630362429694720911 [ 9247.090179] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9247.102354] LustreError: 375156:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9247.505475] Lustre: server umount lustre-MDT0001 complete [ 9255.202444] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 9256.739480] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 04:45:06 (1788338706) [ 9262.195706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9262.563390] LustreError: 379596:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9262.584262] LustreError: 379596:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9262.634757] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9266.704753] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 9267.686446] LustreError: 379597:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9272.801938] LustreError: 379596:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9275.047901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9275.462045] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9279.484719] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 9286.745306] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9291.521330] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9297.077425] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9297.481302] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9303.266935] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 9309.959822] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9315.777920] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9315.956181] Lustre: lustre-OST0001: new disk, initializing [ 9315.962859] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9315.976375] Lustre: 382432:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9316.044935] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9321.704313] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 9324.644149] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9324.651199] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9324.737612] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9330.528231] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9335.271974] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9335.272427] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9335.281477] LustreError: Skipped 1 previous similar message [ 9335.283512] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9335.294165] Lustre: Skipped 1 previous similar message [ 9335.310197] Lustre: Skipped 2 previous similar messages [ 9339.733081] Lustre: server umount lustre-OST0001 complete [ 9340.389714] LustreError: 381419:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9340.406095] LustreError: 381419:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9345.511185] LustreError: 381418:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9345.526184] LustreError: 381418:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9355.746944] LustreError: 381420:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9355.749215] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9355.750126] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9355.750132] Lustre: Skipped 1 previous similar message [ 9355.766896] LustreError: 381420:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9355.818995] Lustre: Skipped 1 previous similar message [ 9360.875222] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9360.882248] Lustre: Skipped 2 previous similar messages [ 9361.682412] Lustre: server umount lustre-OST0000 complete [ 9365.988046] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9366.009802] Lustre: Skipped 1 previous similar message [ 9371.104475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9371.107237] Lustre: Skipped 3 previous similar messages [ 9371.704782] Lustre: server umount lustre-MDT0000 complete [ 9375.447968] LustreError: 379579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788338825 with bad export cookie 14630362429694721793 [ 9375.457969] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9375.459248] LustreError: 379579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9381.945968] Lustre: server umount lustre-MDT0001 complete [ 9391.142956] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 9393.776300] Key type lgssc unregistered [ 9394.086685] LNet: 384418:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9394.093673] LNetError: 384418:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9394.124695] LNet: Removed LNI 192.168.203.160@tcp [ 9395.046573] Key type .llcrypt unregistered [ 9395.049168] Key type ._llcrypt unregistered [ 9442.406334] Key type ._llcrypt registered [ 9442.411393] Key type .llcrypt registered [ 9442.558919] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 9443.551028] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9443.584336] alg: No test for adler32 (adler32-zlib) [ 9444.651961] Lustre: Lustre: Build Version: 2.17.57_104_g6cc0f2c [ 9444.874690] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 9446.567746] Key type lgssc registered [ 9447.526899] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9454.129775] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9454.142561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9455.383437] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9455.405859] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9455.537839] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9459.845488] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 9467.592284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9467.698489] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9467.724456] Lustre: 387193:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9468.022709] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9472.436789] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 9479.754209] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9485.169504] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9488.355975] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9488.362932] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9488.395728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9491.941118] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9491.943427] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9493.169506] Lustre: server umount lustre-MDT0000 complete [ 9496.824834] LustreError: 387185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788338946 with bad export cookie 17646267781960098236 [ 9496.828486] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9496.840271] LustreError: 387185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9497.166988] Lustre: server umount lustre-MDT0001 complete [ 9505.786157] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 04:49:15 (1788338955) [ 9511.364948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9511.826650] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9515.500268] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 9517.033126] LustreError: 389705:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9517.064443] LustreError: 389705:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9522.146628] LustreError: 389706:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9523.052120] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9523.440233] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9527.286524] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 9533.530289] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9538.166870] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9548.964873] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9549.467568] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9549.908584] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9560.116020] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 9572.819158] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9582.044160] Lustre: Permanently deactivating lustre-OST0000 [ 9582.076464] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 9587.319691] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 9588.607772] LustreError: 392599:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 9588.616994] LustreError: 392599:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [ 9588.623808] LustreError: 392599:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 9591.120677] LustreError: 392647:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [ 9591.129869] LustreError: 392647:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 9599.035831] Lustre: server umount lustre-OST0000 complete [ 9601.541594] LustreError: 392896:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [ 9601.545759] LustreError: 392896:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 9605.091820] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9605.096544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9605.109396] Lustre: Skipped 1 previous similar message [ 9605.131957] Lustre: Skipped 2 previous similar messages [ 9610.210895] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9610.219686] Lustre: Skipped 1 previous similar message [ 9613.384730] Lustre: server umount lustre-MDT0000 complete [ 9615.332055] LustreError: 390431:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9615.360501] LustreError: 390431:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9617.295169] LustreError: 392399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788339067 with bad export cookie 17646267781960098817 [ 9617.305554] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9617.311881] LustreError: 392399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9627.874823] Lustre: server umount lustre-MDT0001 complete [ 9637.707306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9642.396849] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 9645.405372] LustreError: 394055:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 9651.764200] Lustre: 394393:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.160@tcp [ 9653.783663] Lustre: server umount MGS complete [ 9665.236851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9665.690584] LustreError: 395117:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9665.718974] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 9665.726996] Lustre: Skipped 1 previous similar message [ 9665.814310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9670.491321] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 9671.140913] LustreError: 395118:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9679.000190] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9679.285297] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 9683.511701] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 9691.339458] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9697.158332] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9699.089207] Lustre: Permanently reactivating lustre-OST0000 [ 9699.118627] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 9699.126580] Lustre: Skipped 2 previous similar messages [ 9728.480371] LustreError: 396950:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 9735.137391] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9735.588758] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9735.593425] Lustre: Skipped 1 previous similar message [ 9742.642668] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 9750.981620] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9758.969447] Lustre: server umount lustre-OST0000 complete [ 9764.320918] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9764.327563] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9764.333257] Lustre: Skipped 1 previous similar message [ 9764.346520] Lustre: Skipped 1 previous similar message [ 9772.584124] Lustre: server umount lustre-MDT0000 complete [ 9774.563261] LustreError: 395848:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9774.580216] LustreError: 395848:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9775.519241] Lustre: 386516:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788339209/real 1788339209] req@00000000fc2086cd x1875209244214784/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788339225 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9775.533779] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9775.932491] LustreError: 395098:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788339225 with bad export cookie 17646267781960099867 [ 9775.951786] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9776.380287] Lustre: server umount lustre-MDT0001 complete [ 9786.120365] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 9788.767447] Key type lgssc unregistered [ 9789.131485] LNet: 399533:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9789.142026] LNetError: 399533:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9789.164238] LNet: Removed LNI 192.168.203.160@tcp [ 9790.158473] Key type .llcrypt unregistered [ 9790.162789] Key type ._llcrypt unregistered [ 9807.919415] Key type ._llcrypt registered [ 9807.922607] Key type .llcrypt registered [ 9808.064597] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_hostid [ 9823.475918] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 9824.349796] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9824.551320] alg: No test for adler32 (adler32-zlib) [ 9825.592212] Lustre: Lustre: Build Version: 2.17.57_104_g6cc0f2c [ 9825.864341] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 9827.543238] Key type lgssc registered [ 9828.697822] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9867.549230] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9867.627509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9868.944497] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9869.019119] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9869.144305] Lustre: lustre-MDT0000: new disk, initializing [ 9869.231756] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9869.272973] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9874.119623] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 9885.334435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9885.494027] Lustre: 402688:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9885.585970] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9885.592125] Lustre: Skipped 1 previous similar message [ 9885.701772] Lustre: lustre-MDT0001: new disk, initializing [ 9885.802185] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9885.850601] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9885.859248] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9890.372603] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 9900.642172] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9906.524432] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9913.550668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9913.922522] Lustre: lustre-OST0000: new disk, initializing [ 9913.926409] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9913.935825] Lustre: 404655:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9914.035146] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9919.443274] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 9920.071754] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9920.080486] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9920.189071] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9927.511712] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9931.941655] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9932.137142] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9936.252676] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9936.587816] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9940.985659] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9940.990521] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9940.993363] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9940.998309] Lustre: Skipped 1 previous similar message [ 9941.004477] LustreError: Skipped 1 previous similar message [ 9944.511706] Lustre: server umount lustre-OST0000 complete [ 9951.202571] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9951.205560] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9951.215779] Lustre: Skipped 1 previous similar message [ 9951.229883] Lustre: Skipped 2 previous similar messages [ 9953.804392] Lustre: server umount lustre-MDT0000 complete [ 9957.439156] LustreError: 402678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788339407 with bad export cookie 5936498651531108773 [ 9957.442886] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9957.450628] LustreError: 402678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9957.921156] Lustre: server umount lustre-MDT0001 complete [ 9966.033649] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 9967.715749] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 04:56:57 (1788339417) [ 9974.605184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9975.121326] LustreError: 407114:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9975.142398] LustreError: 407114:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9975.231257] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9979.081409] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 9980.390668] LustreError: 407115:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9985.504795] LustreError: 407114:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9986.884237] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9987.198591] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9991.974431] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 9999.600887] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10005.030917] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10011.361330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10011.593333] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10017.370509] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [10024.642960] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10026.589085] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [10030.077562] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [10036.197114] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10036.201556] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10036.208202] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10036.217470] Lustre: Skipped 1 previous similar message [10041.024394] Lustre: server umount lustre-OST0000 complete [10046.437699] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10046.464207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10046.469434] Lustre: Skipped 1 previous similar message [10050.643796] Lustre: server umount lustre-MDT0000 complete [10051.554403] LustreError: 407115:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10051.572837] LustreError: 407115:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10053.605699] LustreError: 407097:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788339503 with bad export cookie 5936498651531109655 [10053.610055] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10053.617354] LustreError: 407097:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10053.874568] Lustre: server umount lustre-MDT0001 complete [10063.370773] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [10065.949723] Key type lgssc unregistered [10066.255429] LNet: 410838:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10066.269685] LNetError: 410838:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10066.281175] LNet: Removed LNI 192.168.203.160@tcp [10067.105275] Key type .llcrypt unregistered [10067.107879] Key type ._llcrypt unregistered [10087.451793] Key type ._llcrypt registered [10087.454345] Key type .llcrypt registered [10087.568843] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [10087.792327] Key type .llcrypt unregistered [10087.797563] Key type ._llcrypt unregistered [10102.971487] Key type ._llcrypt registered [10102.974570] Key type .llcrypt registered [10103.064826] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [10104.048203] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10104.105546] alg: No test for adler32 (adler32-zlib) [10105.146634] Lustre: Lustre: Build Version: 2.17.57_104_g6cc0f2c [10105.365315] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [10107.008330] Key type lgssc registered [10107.873748] Lustre: Echo OBD driver; http://www.lustre.org/ [10114.922407] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10114.941315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10116.436368] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10120.989256] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [10126.821745] LustreError: 413034:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10126.849365] LustreError: 413034:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10129.481748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10129.805237] LustreError: 413035:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10129.835033] LustreError: 413035:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10129.842800] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10135.009742] LustreError: 413034:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10135.046958] LustreError: 413034:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10136.466390] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [10140.135205] LustreError: 414054:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10140.164332] LustreError: 414054:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10141.184208] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10144.141652] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10146.278322] LustreError: 413034:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10146.293307] LustreError: 413034:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10151.191525] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10151.570724] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10155.740747] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [10165.116955] Lustre: server umount lustre-OST0000 complete [10170.857702] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [10170.869283] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10170.891526] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10172.386031] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10172.387669] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10177.512122] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10177.519977] Lustre: Skipped 2 previous similar messages [10178.615179] Lustre: server umount lustre-MDT0000 complete [10182.286865] LustreError: 413014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788339632 with bad export cookie 124636543917743553 [10182.299427] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10182.304095] LustreError: 413014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10193.015224] Lustre: server umount lustre-MDT0001 complete [10202.557463] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [10204.836460] Key type lgssc unregistered [10205.067942] LNet: 416183:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10205.074069] LNetError: 416183:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10205.098239] LNet: Removed LNI 192.168.203.160@tcp [10205.775602] Key type .llcrypt unregistered [10205.777517] Key type ._llcrypt unregistered [10230.051996] Key type ._llcrypt registered [10230.055174] Key type .llcrypt registered [10230.181227] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [10231.288025] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10231.307643] alg: No test for adler32 (adler32-zlib) [10232.579180] Lustre: Lustre: Build Version: 2.17.57_104_g6cc0f2c [10233.010487] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [10234.815208] Key type lgssc registered [10236.418656] Lustre: Echo OBD driver; http://www.lustre.org/ [10245.621531] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10245.648762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10261.215429] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [10266.339990] LustreError: 417579:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.160@tcp: failed processing log, type 4: rc = -110 [10292.960183] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10298.752991] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [10306.572525] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10314.121680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10314.543512] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10314.643183] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10319.248531] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [10319.844929] LustreError: 417604:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10319.874662] LustreError: 417604:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10324.964296] LustreError: 417605:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10327.623762] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10328.057105] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10331.984896] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [10339.453373] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10345.571968] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10353.632074] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10353.638620] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10353.649960] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10354.530944] Lustre: server umount lustre-OST0000 complete [10358.756651] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10358.768903] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10358.788582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10359.782816] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10359.797193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10359.803687] Lustre: Skipped 1 previous similar message [10364.899937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10364.916185] Lustre: Skipped 1 previous similar message [10368.054701] Lustre: server umount lustre-MDT0000 complete [10370.019909] LustreError: 418618:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10370.037032] LustreError: 418618:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10371.702390] LustreError: 417585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788339821 with bad export cookie 13443536700680798357 [10371.712142] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10371.720708] LustreError: 417585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10372.085845] Lustre: server umount lustre-MDT0001 complete [10381.848902] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [10384.477154] Key type lgssc unregistered [10384.736626] LNet: 421148:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10384.749554] LNetError: 421148:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10384.763668] LNet: Removed LNI 192.168.203.160@tcp [10385.478943] Key type .llcrypt unregistered [10385.482682] Key type ._llcrypt unregistered [10409.301896] Key type ._llcrypt registered [10409.306049] Key type .llcrypt registered [10409.448529] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [10411.057080] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10411.104258] alg: No test for adler32 (adler32-zlib) [10412.275286] Lustre: Lustre: Build Version: 2.17.57_104_g6cc0f2c [10412.577375] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [10414.264222] Key type lgssc registered [10415.457315] Lustre: Echo OBD driver; http://www.lustre.org/ [10422.728616] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10422.746792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10424.275259] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10428.525881] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [10434.528951] LustreError: 422563:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10434.542866] LustreError: 422563:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10436.636305] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10436.963716] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10441.626149] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [10449.005474] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10454.934473] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10462.161776] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10462.470765] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10468.573233] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [10469.695651] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10476.135137] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10481.957706] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10481.966267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10481.972345] Lustre: Skipped 1 previous similar message [10483.938932] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.60@tcp (stopping) [10483.951414] Lustre: Skipped 2 previous similar messages [10487.275816] LustreError: 423291:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10487.300462] LustreError: 423291:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10487.390759] Lustre: server umount lustre-MDT0000 complete [10489.014936] LustreError: 422558:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10489.954224] LustreError: 422544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.60@tcp arrived at 1788339939 with bad export cookie 17656568465013566588 [10489.982695] LustreError: 422544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10491.543932] LustreError: 422542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788339941 with bad export cookie 17656568465013565699 [10491.556229] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10492.384938] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10492.386702] LustreError: 422558:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10492.400481] Lustre: Skipped 1 previous similar message [10492.402491] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10492.429709] LustreError: 422558:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10496.416462] LustreError: 422563:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10496.416670] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10496.426217] LustreError: 422563:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10497.784925] Lustre: server umount lustre-OST0000 complete [10504.543583] LustreError: 423293:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000dbabc3e8 x1875210259169536/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10504.698887] Lustre: server umount lustre-MDT0001 complete [10514.847598] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [10517.490316] Key type lgssc unregistered [10517.842691] LNet: 426248:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10517.851791] LNetError: 426248:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10517.875867] LNet: Removed LNI 192.168.203.160@tcp [10518.614700] Key type .llcrypt unregistered [10518.616787] Key type ._llcrypt unregistered [10546.032404] Key type ._llcrypt registered [10546.041537] Key type .llcrypt registered [10546.160445] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [10547.466576] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10547.517235] alg: No test for adler32 (adler32-zlib) [10548.764507] Lustre: Lustre: Build Version: 2.17.57_104_g6cc0f2c [10549.184265] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [10550.959177] Key type lgssc registered [10552.154357] Lustre: Echo OBD driver; http://www.lustre.org/ [10558.883386] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10558.920150] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10560.324319] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10564.785792] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [10570.723191] LustreError: 427661:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10570.742975] LustreError: 427661:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10573.351580] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10573.780798] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10578.465411] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [10585.648314] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10592.307667] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10600.634583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10601.256557] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10605.367087] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10609.086209] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [10616.479815] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10623.969852] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10623.995383] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10627.886875] Lustre: server umount lustre-MDT0001 complete [10629.094545] LustreError: 427662:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10629.115181] LustreError: 427662:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10634.222722] LustreError: 428389:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10634.231767] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10634.251868] LustreError: 428389:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10634.283144] Lustre: Skipped 1 previous similar message [10634.290081] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10634.293787] Lustre: Skipped 2 previous similar messages [10639.329675] LustreError: 427661:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10639.344214] Lustre: server umount lustre-OST0000 complete [10643.473583] Lustre: server umount lustre-MDT0000 complete [10658.137426] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [10661.248047] Key type lgssc unregistered [10661.527438] LNet: 431345:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10661.531166] LNetError: 431345:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10661.543554] LNet: Removed LNI 192.168.203.160@tcp [10662.525804] Key type .llcrypt unregistered [10662.531454] Key type ._llcrypt unregistered [10687.537839] Key type ._llcrypt registered [10687.540623] Key type .llcrypt registered [10687.670186] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_hostid [10703.700541] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [10704.882368] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10704.993071] alg: No test for adler32 (adler32-zlib) [10706.041522] Lustre: Lustre: Build Version: 2.17.57_104_g6cc0f2c [10706.348679] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [10708.063157] Key type lgssc registered [10709.135680] Lustre: Echo OBD driver; http://www.lustre.org/ [10743.450232] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [10756.737446] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10756.758230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10757.969792] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10758.000635] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10758.114470] Lustre: lustre-MDT0000: new disk, initializing [10758.190650] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10758.206104] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10762.210419] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [10775.404737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10775.779265] Lustre: lustre-OST0000: new disk, initializing [10775.784929] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10775.795209] Lustre: Skipped 1 previous similar message [10775.807448] Lustre: 436378:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10775.904494] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10781.133893] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [10781.796961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [10781.804144] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [10781.849696] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [10789.572996] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10803.119453] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10803.330106] Lustre: 435359:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10803.426219] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10803.562622] Lustre: lustre-MDT0001: new disk, initializing [10803.780481] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10803.830195] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [10803.849636] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [10808.079207] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [10817.196803] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [10817.356768] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [10821.325810] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [10821.575620] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [10827.469990] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.60@tcp (stopping) [10828.266538] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10828.268174] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [10828.269355] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10828.278881] Lustre: Skipped 1 previous similar message [10832.763843] Lustre: server umount lustre-MDT0001 complete [10838.501163] LustreError: 435371:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10838.517530] LustreError: 435371:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10843.622918] LustreError: 437090:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10843.643134] LustreError: 437090:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10848.739582] LustreError: 435365:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10848.762351] LustreError: 435365:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10853.344159] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10853.349295] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10853.363885] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10853.369561] Lustre: Skipped 1 previous similar message [10853.861790] LustreError: 435370:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10856.181828] Lustre: server umount lustre-OST0000 complete [10858.980168] LustreError: 435371:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10865.590691] Lustre: server umount lustre-MDT0000 complete [10876.481254] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [10879.069936] Key type lgssc unregistered [10879.322029] LNet: 440120:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10879.330225] LNetError: 440120:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10879.346824] LNet: Removed LNI 192.168.203.160@tcp [10880.016682] Key type .llcrypt unregistered [10880.018762] Key type ._llcrypt unregistered [10916.613896] Key type ._llcrypt registered [10916.616948] Key type .llcrypt registered [10916.716386] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [10916.969313] Key type .llcrypt unregistered [10916.974146] Key type ._llcrypt unregistered [10932.425931] Key type ._llcrypt registered [10932.428937] Key type .llcrypt registered [10932.554966] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [10933.483281] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10933.619367] alg: No test for adler32 (adler32-zlib) [10934.805265] Lustre: Lustre: Build Version: 2.17.57_104_g6cc0f2c [10935.159415] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [10936.839162] Key type lgssc registered [10937.800291] Lustre: Echo OBD driver; http://www.lustre.org/ [10960.037130] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10960.074953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10961.285991] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10961.309190] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10961.375712] Lustre: lustre-MDT0000: new disk, initializing [10961.433550] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10961.448389] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10964.833484] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [10972.895990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10972.986848] Lustre: 443487:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10973.044496] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10973.051251] Lustre: Skipped 1 previous similar message [10973.114729] Lustre: lustre-MDT0001: new disk, initializing [10973.183732] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10973.213900] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10973.224196] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10976.510796] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [10983.588579] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10987.903734] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10992.969033] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10993.187772] Lustre: lustre-OST0000: new disk, initializing [10993.192502] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10993.198619] Lustre: 445454:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10993.267953] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10998.741652] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [10999.356460] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10999.365522] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10999.487831] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11006.674830] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11014.113380] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11014.122736] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11014.140491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11015.140686] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11015.145918] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11015.161472] Lustre: Skipped 1 previous similar message [11018.454814] Lustre: server umount lustre-MDT0000 complete [11021.732618] LustreError: 443480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788340471 with bad export cookie 5190494496962959877 [11021.740245] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11021.741865] LustreError: 443480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11022.095850] Lustre: server umount lustre-MDT0001 complete [11035.780493] Lustre: server umount lustre-OST0000 complete [11044.344545] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 05:14:53 (1788340493) [11050.409070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11050.717895] LustreError: 447484:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11050.733434] LustreError: 447484:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [11050.779494] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11053.953853] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [11056.101712] LustreError: 447485:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11060.194567] LustreError: 447484:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11061.198553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11061.570641] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11065.074456] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [11071.156432] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11075.513611] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11093.548711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11093.675162] Lustre: Found index 0 for lustre-OST0000, updating log [11093.807024] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11096.876307] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [11098.562699] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [11104.173586] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11112.416294] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11112.419801] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11112.422290] LustreError: Skipped 1 previous similar message [11112.426829] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11112.435125] Lustre: Skipped 1 previous similar message [11112.439143] Lustre: Skipped 2 previous similar messages [11113.907760] Lustre: server umount lustre-OST0000 complete [11129.824954] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11129.829339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11129.837113] Lustre: Skipped 1 previous similar message [11129.849499] Lustre: Skipped 4 previous similar messages [11134.877418] Lustre: server umount lustre-MDT0000 complete [11134.946459] LustreError: 447485:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11134.953900] LustreError: 447485:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11137.451574] LustreError: 447466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788340587 with bad export cookie 5190494496962961270 [11137.455265] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11137.460631] LustreError: 447466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11137.713709] Lustre: server umount lustre-MDT0001 complete [11144.552125] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 05:16:34 (1788340594) [11159.455379] LNet: 1 local NIs in recovery (showing 1): 192.168.203.160@tcp [11742.374256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11742.630467] LustreError: 453885:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11742.672684] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11745.252906] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [11750.370751] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11750.601102] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11753.185463] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [11757.842863] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11761.217541] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11778.278657] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11778.407684] Lustre: Found index 0 for lustre-OST0000, updating log [11778.555080] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11781.944611] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [11786.456875] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11787.731196] Lustre: server umount lustre-OST0000 complete [11791.840931] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11791.848593] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11791.860295] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11793.890582] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11793.891897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11793.903011] Lustre: Skipped 1 previous similar message [11796.211384] Lustre: server umount lustre-MDT0000 complete [11797.985608] LustreError: 453882:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11797.996659] LustreError: 453882:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11798.204470] LustreError: 453867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788341248 with bad export cookie 5190494496962962348 [11798.206327] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11798.211620] LustreError: 453867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11798.379427] Lustre: server umount lustre-MDT0001 complete [11800.479362] LNet: 1 local NIs in recovery (showing 1): 192.168.203.160@tcp [11821.442639] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_hostid [11827.833251] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [11852.929604] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 05:28:22 (1788341302) [11863.997424] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [11865.695700] Key type lgssc unregistered [11865.872642] LNet: 460489:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11865.879716] LNetError: 460489:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11908.902149] LNet: Removed LNI 192.168.203.160@tcp [11909.338122] Key type .llcrypt unregistered [11909.339751] Key type ._llcrypt unregistered [11918.458146] Key type ._llcrypt registered [11918.460894] Key type .llcrypt registered [11918.513950] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [11918.942233] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11919.032254] alg: No test for adler32 (adler32-zlib) [11919.960713] Lustre: Lustre: Build Version: 2.17.57_104_g6cc0f2c [11920.093211] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [11921.695185] Key type lgssc registered [11922.419393] Lustre: Echo OBD driver; http://www.lustre.org/ [11924.768806] LNet: Added LNI 192.168.203.160@tcp100 [8/256/0/180] [11925.280552] LNet: Added LNI 192.168.203.160@tcp101 [8/256/0/180] [11925.769612] LNet: Added LNI 192.168.203.160@tcp102 [8/256/0/180] [11926.280922] LNet: Added LNI 192.168.203.160@tcp103 [8/256/0/180] [11926.740724] LNet: Added LNI 192.168.203.160@tcp104 [8/256/0/180] [11927.191042] LNet: Added LNI 192.168.203.160@tcp105 [8/256/0/180] [11927.627381] LNet: Added LNI 192.168.203.160@tcp106 [8/256/0/180] [11928.095558] LNet: Added LNI 192.168.203.160@tcp107 [8/256/0/180] [11928.554119] LNet: Added LNI 192.168.203.160@tcp108 [8/256/0/180] [11928.993952] LNet: Added LNI 192.168.203.160@tcp109 [8/256/0/180] [11929.437853] LNet: Added LNI 192.168.203.160@tcp110 [8/256/0/180] [11929.874859] LNet: Added LNI 192.168.203.160@tcp111 [8/256/0/180] [11930.327318] LNet: Added LNI 192.168.203.160@tcp112 [8/256/0/180] [11930.771526] LNet: Added LNI 192.168.203.160@tcp113 [8/256/0/180] [11931.223050] LNet: Added LNI 192.168.203.160@tcp114 [8/256/0/180] [11931.664812] LNet: Added LNI 192.168.203.160@tcp115 [8/256/0/180] [11932.157956] LNet: Added LNI 192.168.203.160@tcp116 [8/256/0/180] [11932.633220] LNet: Added LNI 192.168.203.160@tcp117 [8/256/0/180] [11933.191356] LNet: Added LNI 192.168.203.160@tcp118 [8/256/0/180] [11933.671743] LNet: Added LNI 192.168.203.160@tcp119 [8/256/0/180] [11934.145780] LNet: Added LNI 192.168.203.160@tcp120 [8/256/0/180] [11934.680790] LNet: Added LNI 192.168.203.160@tcp121 [8/256/0/180] [11935.213983] LNet: Added LNI 192.168.203.160@tcp122 [8/256/0/180] [11935.752334] LNet: Added LNI 192.168.203.160@tcp123 [8/256/0/180] [11936.352984] LNet: Added LNI 192.168.203.160@tcp124 [8/256/0/180] [11936.946117] LNet: Added LNI 192.168.203.160@tcp125 [8/256/0/180] [11937.579571] LNet: Added LNI 192.168.203.160@tcp126 [8/256/0/180] [11938.261367] LNet: Added LNI 192.168.203.160@tcp127 [8/256/0/180] [11938.910912] LNet: Added LNI 192.168.203.160@tcp128 [8/256/0/180] [11939.491342] LNet: Added LNI 192.168.203.160@tcp129 [8/256/0/180] [11940.048752] LNet: Added LNI 192.168.203.160@tcp130 [8/256/0/180] [11940.574658] LNet: Added LNI 192.168.203.160@tcp131 [8/256/0/180] [11941.126427] LNet: Added LNI 192.168.203.160@tcp132 [8/256/0/180] [11941.804985] LNet: Added LNI 192.168.203.160@tcp133 [8/256/0/180] [11942.359424] LNet: Added LNI 192.168.203.160@tcp134 [8/256/0/180] [11942.973546] LNet: Added LNI 192.168.203.160@tcp135 [8/256/0/180] [11947.356676] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11947.374848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11948.455691] Lustre: 463481:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.160@tcp: can't add NID 192.168.203.160@tcp130: rc = -75 [11948.507815] LNetError: 460944:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.160 [11948.516681] LNetError: Protocol error connecting to 192.168.203.160@tcp121 at host 192.168.203.160:988: is it running a compatible version of Lustre? [11948.529062] LNetError: Skipped 6 previous similar messages [11948.566247] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11948.594436] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11948.653700] Lustre: lustre-MDT0000: new disk, initializing [11948.705374] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11948.726589] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11948.767165] LNet: 6 local NIs in recovery (showing 6): 192.168.203.160@tcp125, 192.168.203.160@tcp126, 192.168.203.160@tcp127, 192.168.203.160@tcp129, 192.168.203.160@tcp128, 192.168.203.160@tcp121 [11949.536403] LNetError: 463501:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.160 [11949.545420] LNetError: 463501:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 7 previous similar messages [11949.550242] LNetError: Protocol error connecting to 192.168.203.160@tcp at host 192.168.203.160:988: is it running a compatible version of Lustre? [11951.470182] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [11951.842771] LNetError: 463504:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.160 [11951.848946] LNetError: 463504:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 6 previous similar messages [11951.855928] LNetError: Protocol error connecting to 192.168.203.160@tcp129 at host 192.168.203.160:988: is it running a compatible version of Lustre? [11951.862548] LNetError: Skipped 7 previous similar messages [11955.936944] LNetError: 463514:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.160 [11955.949347] LNetError: 463514:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [11955.955353] LNetError: Protocol error connecting to 192.168.203.160@tcp129 at host 192.168.203.160:988: is it running a compatible version of Lustre? [11958.895782] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11958.979484] Lustre: 463496:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11959.022945] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11959.027228] Lustre: Skipped 1 previous similar message [11959.088235] Lustre: lustre-MDT0001: new disk, initializing [11959.106139] Lustre: 464300:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.160@tcp: can't add NID 192.168.203.160@tcp130: rc = -75 [11959.112878] Lustre: 464300:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [11959.188752] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11959.225741] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11959.236056] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11961.533243] Lustre: 464511:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.160@tcp: can't add NID 192.168.203.160@tcp130: rc = -75 [11961.542591] Lustre: 464511:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [11961.840471] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [11963.104979] LNetError: 460943:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.160 [11963.116496] LNetError: Protocol error connecting to 192.168.203.160@tcp129 at host 192.168.203.160:988: is it running a compatible version of Lustre? [11967.907980] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11971.488161] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11975.828900] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11976.010043] Lustre: lustre-OST0000: new disk, initializing [11976.012268] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11976.016617] Lustre: 465471:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11976.075547] Lustre: 465476:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.160@tcp: can't add NID 192.168.203.160@tcp130: rc = -75 [11976.080106] Lustre: 465476:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [11976.115068] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11979.489133] LNetError: 463501:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.160 [11979.494798] LNetError: Protocol error connecting to 192.168.203.160@tcp129 at host 192.168.203.160:988: is it running a compatible version of Lustre? [11979.826562] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [11981.831204] Lustre: 465928:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.160@tcp: can't add NID 192.168.203.160@tcp130: rc = -75 [11981.838744] Lustre: 465928:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [11981.870573] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11981.877251] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11981.973280] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11986.482318] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11992.550544] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11992.553822] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11992.583276] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11995.872470] LNetError: 460944:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.160 [11995.879795] LNetError: 460944:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 5 previous similar messages [11995.888774] LNetError: Protocol error connecting to 192.168.203.160@tcp124 at host 192.168.203.160:988: is it running a compatible version of Lustre? [11995.896179] LNetError: Skipped 5 previous similar messages [11997.324939] Lustre: server umount lustre-OST0000 complete [12000.226240] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12000.233689] LustreError: Skipped 1 previous similar message [12000.237898] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12000.255543] Lustre: Skipped 1 previous similar message [12000.263342] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12000.270620] Lustre: Skipped 1 previous similar message [12002.786719] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12002.787939] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12005.920954] Lustre: server umount lustre-MDT0000 complete [12008.161682] LNet: 1 local NIs in recovery (showing 1): 192.168.203.160@tcp129 [12008.507340] LustreError: 464299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788341458 with bad export cookie 14229408891091262684 [12008.511439] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12008.520589] LustreError: 464299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12008.706970] Lustre: server umount lustre-MDT0001 complete [12016.741546] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [12018.974819] Key type lgssc unregistered [12019.311981] LNet: 467380:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12019.319237] LNetError: 467380:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12019.324605] LNet: Removed LNI 192.168.203.160@tcp [12020.383308] LNet: Removed LNI 192.168.203.160@tcp100 [12020.386916] LNetError: 467380:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12020.392270] LNetError: 467380:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [12020.398547] LNet: Removed LNI 192.168.203.160@tcp101 [12021.407265] LNet: Removed LNI 192.168.203.160@tcp102 [12021.412777] LNetError: 467380:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12021.425542] LNetError: 467380:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [12021.434441] LNet: Removed LNI 192.168.203.160@tcp103 [12022.495251] LNet: Removed LNI 192.168.203.160@tcp104 [12023.519385] LNet: Removed LNI 192.168.203.160@tcp105 [12023.526540] LNetError: 467380:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12023.534837] LNetError: 467380:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [12023.539945] LNet: Removed LNI 192.168.203.160@tcp106 [12024.543233] LNet: Removed LNI 192.168.203.160@tcp107 [12025.567242] LNet: Removed LNI 192.168.203.160@tcp108 [12026.591255] LNet: Removed LNI 192.168.203.160@tcp109 [12027.615329] LNet: Removed LNI 192.168.203.160@tcp110 [12027.620901] LNetError: 467380:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12027.629462] LNetError: 467380:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [12027.638079] LNet: Removed LNI 192.168.203.160@tcp111 [12028.703390] LNet: Removed LNI 192.168.203.160@tcp112 [12029.727271] LNet: Removed LNI 192.168.203.160@tcp113 [12030.751176] LNet: Removed LNI 192.168.203.160@tcp114 [12031.775194] LNet: Removed LNI 192.168.203.160@tcp115 [12032.800476] LNet: Removed LNI 192.168.203.160@tcp116 [12033.823176] LNet: Removed LNI 192.168.203.160@tcp117 [12034.847420] LNet: Removed LNI 192.168.203.160@tcp118 [12035.871574] LNet: Removed LNI 192.168.203.160@tcp119 [12035.876315] LNetError: 467380:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12035.880652] LNetError: 467380:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [12035.884496] LNet: Removed LNI 192.168.203.160@tcp120 [12036.895208] LNet: Removed LNI 192.168.203.160@tcp121 [12037.919802] LNet: Removed LNI 192.168.203.160@tcp122 [12038.943397] LNet: Removed LNI 192.168.203.160@tcp123 [12039.967313] LNet: Removed LNI 192.168.203.160@tcp124 [12040.992053] LNet: Removed LNI 192.168.203.160@tcp125 [12042.015907] LNet: Removed LNI 192.168.203.160@tcp126 [12043.039873] LNet: Removed LNI 192.168.203.160@tcp127 [12044.063140] LNet: Removed LNI 192.168.203.160@tcp128 [12044.067569] LNet: Removed LNI 192.168.203.160@tcp129 [12044.071707] LNet: Removed LNI 192.168.203.160@tcp130 [12044.076374] LNet: Removed LNI 192.168.203.160@tcp131 [12044.080175] LNet: Removed LNI 192.168.203.160@tcp132 [12044.083885] LNet: Removed LNI 192.168.203.160@tcp133 [12044.086808] LNet: Removed LNI 192.168.203.160@tcp134 [12044.097078] LNet: Removed LNI 192.168.203.160@tcp135 [12044.614205] Key type .llcrypt unregistered [12044.615942] Key type ._llcrypt unregistered [12057.765491] Key type ._llcrypt registered [12057.767550] Key type .llcrypt registered [12057.834366] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_hostid [12066.617534] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [12067.252144] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12067.280527] alg: No test for adler32 (adler32-zlib) [12068.203207] Lustre: Lustre: Build Version: 2.17.57_104_g6cc0f2c [12068.378520] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [12069.991158] Key type lgssc registered [12070.571431] Lustre: Echo OBD driver; http://www.lustre.org/ [12089.069834] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 05:32:18 (1788341538) [12098.611092] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [12100.179538] Key type lgssc unregistered [12100.374802] LNet: 471438:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12100.379189] LNetError: 471438:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12100.394607] LNet: Removed LNI 192.168.203.160@tcp [12100.889434] Key type .llcrypt unregistered [12100.891266] Key type ._llcrypt unregistered [12110.456650] Key type ._llcrypt registered [12110.458273] Key type .llcrypt registered [12110.513407] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [12110.990920] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12111.020945] alg: No test for adler32 (adler32-zlib) [12111.966161] Lustre: Lustre: Build Version: 2.17.57_104_g6cc0f2c [12112.144206] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [12113.767352] Key type lgssc registered [12114.324269] Lustre: Echo OBD driver; http://www.lustre.org/ [12119.704298] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12119.721330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12120.839900] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12120.860894] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12120.909283] Lustre: lustre-MDT0000: new disk, initializing [12120.963486] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12120.980242] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12123.363859] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [12130.112043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12130.186789] Lustre: 472623:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12130.223562] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12130.226325] Lustre: Skipped 1 previous similar message [12130.279489] Lustre: lustre-MDT0001: new disk, initializing [12130.339293] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12130.366480] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12130.372338] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12132.943855] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [12138.606435] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12141.804784] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12145.464541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12145.624533] Lustre: lustre-OST0000: new disk, initializing [12145.626483] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [12145.631690] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12145.635810] Lustre: 474592:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12145.675486] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12148.799875] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [12150.790886] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12150.796495] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12150.814310] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [12154.859565] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12157.219518] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing lnet_if_list [12160.259371] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [12160.286383] LNet: Added LNI 192.168.203.160@tcp10 [8/256/0/180] [12163.979197] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [12163.996015] LNet: Added LNI 192.168.203.160@tcp11 [8/256/0/180] [12166.956479] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [12166.983020] LNet: Added LNI 192.168.203.160@tcp12 [8/256/0/180] [12170.877518] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [12170.908608] LNet: Added LNI 192.168.203.160@tcp13 [8/256/0/180] [12176.015920] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [12176.037067] LNet: Added LNI 192.168.203.160@tcp14 [8/256/0/180] [12182.241964] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [12182.259935] LNet: Added LNI 192.168.203.160@tcp15 [8/256/0/180] [12188.007601] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [12188.040781] LNet: Added LNI 192.168.203.160@tcp16 [8/256/0/180] [12195.259713] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [12195.371253] LNet: Added LNI 192.168.203.160@tcp17 [8/256/0/180] [12204.541320] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [12204.599229] LNet: Added LNI 192.168.203.160@tcp18 [8/256/0/180] [12210.047882] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [12210.162476] LNet: Added LNI 192.168.203.160@tcp19 [8/256/0/180] [12224.416987] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [12224.483255] LNet: Added LNI 192.168.203.160@tcp20 [8/256/0/180] [12238.516851] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [12238.588131] LNet: Added LNI 192.168.203.160@tcp21 [8/256/0/180] [12247.204892] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [12247.270354] LNet: Added LNI 192.168.203.160@tcp22 [8/256/0/180] [12255.911870] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [12255.941838] LNet: Added LNI 192.168.203.160@tcp23 [8/256/0/180] [12263.404150] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [12263.460278] LNet: Added LNI 192.168.203.160@tcp24 [8/256/0/180] [12270.429313] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [12270.469050] LNet: Added LNI 192.168.203.160@tcp25 [8/256/0/180] [12280.228612] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [12280.284588] LNet: Added LNI 192.168.203.160@tcp26 [8/256/0/180] [12289.733534] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [12289.821893] LNet: Added LNI 192.168.203.160@tcp27 [8/256/0/180] [12301.061986] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [12301.109816] LNet: Added LNI 192.168.203.160@tcp28 [8/256/0/180] [12311.114557] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [12311.245646] LNet: Added LNI 192.168.203.160@tcp29 [8/256/0/180] [12319.936469] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [12319.958821] LNet: Added LNI 192.168.203.160@tcp30 [8/256/0/180] [12326.229933] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [12326.288900] LNet: Added LNI 192.168.203.160@tcp31 [8/256/0/180] [12332.099326] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [12332.121968] LNet: Added LNI 192.168.203.160@tcp32 [8/256/0/180] [12337.520442] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [12337.539376] LNet: Added LNI 192.168.203.160@tcp33 [8/256/0/180] [12342.977930] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [12343.010037] LNet: Added LNI 192.168.203.160@tcp34 [8/256/0/180] [12348.360389] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [12348.399613] LNet: Added LNI 192.168.203.160@tcp35 [8/256/0/180] [12354.047559] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [12354.070415] LNet: Added LNI 192.168.203.160@tcp36 [8/256/0/180] [12359.669199] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [12359.703264] LNet: Added LNI 192.168.203.160@tcp37 [8/256/0/180] [12365.318827] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [12365.344218] LNet: Added LNI 192.168.203.160@tcp38 [8/256/0/180] [12371.131354] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [12371.152111] LNet: Added LNI 192.168.203.160@tcp39 [8/256/0/180] [12377.476805] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [12377.522523] LNet: Added LNI 192.168.203.160@tcp40 [8/256/0/180] [12383.699286] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [12383.724126] LNet: Added LNI 192.168.203.160@tcp41 [8/256/0/180] [12390.053328] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [12390.073077] LNet: Added LNI 192.168.203.160@tcp42 [8/256/0/180] [12395.998256] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [12396.030934] LNet: Added LNI 192.168.203.160@tcp43 [8/256/0/180] [12402.160376] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [12402.186764] LNet: Added LNI 192.168.203.160@tcp44 [8/256/0/180] [12408.609399] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [12408.644970] LNet: Added LNI 192.168.203.160@tcp45 [8/256/0/180] [12414.848380] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [12414.874585] LNet: Added LNI 192.168.203.160@tcp46 [8/256/0/180] [12420.832152] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [12420.858160] LNet: Added LNI 192.168.203.160@tcp47 [8/256/0/180] [12427.411510] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [12427.436493] LNet: Added LNI 192.168.203.160@tcp48 [8/256/0/180] [12433.614912] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [12433.656527] LNet: Added LNI 192.168.203.160@tcp49 [8/256/0/180] [12439.361366] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [12439.386462] LNet: Added LNI 192.168.203.160@tcp50 [8/256/0/180] [12453.345826] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12453.352854] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12453.378819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12454.673766] Lustre: server umount lustre-MDT0000 complete [12456.932605] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12457.397850] LustreError: 473418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788341907 with bad export cookie 13835678948039598670 [12457.399567] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12457.409881] LustreError: 473418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12457.647304] Lustre: server umount lustre-MDT0001 complete [12471.326782] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [12477.408417] Lustre: 471899:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788341911/real 1788341912] req@00000000501c9585 x1875212041887744/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788341927 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12477.433730] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12477.446315] Lustre: Skipped 1 previous similar message [12478.253786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12482.596540] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xc00234b2dc402c7a [12482.608766] Lustre: MGC192.168.203.160@tcp: Connection restored to 0@lo (at 0@lo) [12482.799757] LustreError: 474583:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12482.819221] LustreError: 474583:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12482.853604] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12483.871095] Lustre: 471897:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788341917/real 1788341917] req@00000000441fc80a x1875212041888000/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788341933 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12485.898603] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [12487.970415] LustreError: 498807:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12487.984636] LustreError: 498807:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12488.463143] Lustre: 471896:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788341922/real 1788341922] req@000000009aa21890 x1875212041888128/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788341938 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12492.015129] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12492.063169] Lustre: 471898:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788341926/real 1788341926] req@0000000079ef1d6a x1875212041888384/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788341942 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12492.268550] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12495.180923] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [12496.358080] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12500.560873] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12504.245506] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12517.858583] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12517.865660] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12517.875444] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12518.881645] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12521.961826] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12521.971084] Lustre: Skipped 2 previous similar messages [12523.257154] Lustre: server umount lustre-OST0000 complete [12527.072995] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12527.076228] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12527.082339] Lustre: Skipped 2 previous similar messages [12527.097941] Lustre: Skipped 1 previous similar message [12537.312742] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12537.317833] Lustre: Skipped 3 previous similar messages [12540.383197] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12540.534594] Lustre: server umount lustre-MDT0000 complete [12542.432350] LustreError: 498810:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12542.442260] LustreError: 498810:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12542.913069] LustreError: 474590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788341992 with bad export cookie 13835678948039601274 [12542.915692] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12542.930942] LustreError: 474590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12543.146212] Lustre: server umount lustre-MDT0001 complete [12550.017130] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [12551.861108] Key type lgssc unregistered [12552.127271] LNet: 501583:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12552.134443] LNetError: 501583:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12553.183239] LNet: Removed LNI 192.168.203.160@tcp [12553.188720] LNetError: 501583:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12553.199781] LNet: Removed LNI 192.168.203.160@tcp10 [12553.203224] LNet: Removed LNI 192.168.203.160@tcp11 [12553.205843] LNet: Removed LNI 192.168.203.160@tcp12 [12553.209506] LNet: Removed LNI 192.168.203.160@tcp13 [12553.212653] LNet: Removed LNI 192.168.203.160@tcp14 [12553.214651] LNet: Removed LNI 192.168.203.160@tcp15 [12553.217246] LNet: Removed LNI 192.168.203.160@tcp16 [12553.219076] LNet: Removed LNI 192.168.203.160@tcp17 [12553.221178] LNet: Removed LNI 192.168.203.160@tcp18 [12553.223427] LNet: Removed LNI 192.168.203.160@tcp19 [12553.229223] LNet: Removed LNI 192.168.203.160@tcp20 [12553.235125] LNet: Removed LNI 192.168.203.160@tcp21 [12553.241753] LNet: Removed LNI 192.168.203.160@tcp22 [12553.245455] LNet: Removed LNI 192.168.203.160@tcp23 [12553.248455] LNet: Removed LNI 192.168.203.160@tcp24 [12553.252755] LNet: Removed LNI 192.168.203.160@tcp25 [12553.256818] LNet: Removed LNI 192.168.203.160@tcp26 [12553.260246] LNet: Removed LNI 192.168.203.160@tcp27 [12553.264278] LNet: Removed LNI 192.168.203.160@tcp28 [12553.266609] LNet: Removed LNI 192.168.203.160@tcp29 [12553.268898] LNet: Removed LNI 192.168.203.160@tcp30 [12553.271320] LNet: Removed LNI 192.168.203.160@tcp31 [12553.274863] LNet: Removed LNI 192.168.203.160@tcp32 [12553.279145] LNet: Removed LNI 192.168.203.160@tcp33 [12553.281852] LNet: Removed LNI 192.168.203.160@tcp34 [12553.283973] LNet: Removed LNI 192.168.203.160@tcp35 [12553.287051] LNet: Removed LNI 192.168.203.160@tcp36 [12553.289606] LNet: Removed LNI 192.168.203.160@tcp37 [12553.292073] LNet: Removed LNI 192.168.203.160@tcp38 [12553.294655] LNet: Removed LNI 192.168.203.160@tcp39 [12553.297204] LNet: Removed LNI 192.168.203.160@tcp40 [12553.299434] LNet: Removed LNI 192.168.203.160@tcp41 [12553.302711] LNet: Removed LNI 192.168.203.160@tcp42 [12553.305684] LNet: Removed LNI 192.168.203.160@tcp43 [12553.307785] LNet: Removed LNI 192.168.203.160@tcp44 [12553.313095] LNet: Removed LNI 192.168.203.160@tcp45 [12553.318430] LNet: Removed LNI 192.168.203.160@tcp46 [12553.325344] LNet: Removed LNI 192.168.203.160@tcp47 [12553.328581] LNet: Removed LNI 192.168.203.160@tcp48 [12553.330818] LNet: Removed LNI 192.168.203.160@tcp49 [12553.346123] LNet: Removed LNI 192.168.203.160@tcp50 [12553.931364] Key type .llcrypt unregistered [12553.934468] Key type ._llcrypt unregistered [12567.148202] Key type ._llcrypt registered [12567.151063] Key type .llcrypt registered [12567.219653] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_hostid [12576.507982] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [12577.045782] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12577.072618] alg: No test for adler32 (adler32-zlib) [12578.040347] Lustre: Lustre: Build Version: 2.17.57_104_g6cc0f2c [12578.227815] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [12579.855122] Key type lgssc registered [12580.489174] Lustre: Echo OBD driver; http://www.lustre.org/ [12599.330410] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12599.345329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12600.481184] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12600.503687] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12600.558962] Lustre: lustre-MDT0000: new disk, initializing [12600.616340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12600.630946] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12602.817509] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [12608.352777] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12608.409266] Lustre: 504753:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12608.439908] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12608.443412] Lustre: Skipped 1 previous similar message [12608.484525] Lustre: lustre-MDT0001: new disk, initializing [12608.530819] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12608.548848] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12608.558341] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12610.563726] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [12616.056575] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12618.791475] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12622.515620] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12622.626228] Lustre: *** cfs_fail_loc=90f, val=0*** [12622.667442] Lustre: lustre-OST0000: new disk, initializing [12622.672463] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12622.678434] Lustre: 506770:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12622.725509] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12625.695650] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [12628.481042] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12628.485133] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12628.537993] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12630.463158] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12638.688599] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12638.692486] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12638.695362] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12638.705056] Lustre: Skipped 1 previous similar message [12639.851386] Lustre: server umount lustre-OST0000 complete [12643.809809] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12643.818440] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12643.818884] Lustre: Skipped 1 previous similar message [12643.823016] Lustre: Skipped 1 previous similar message [12647.765970] Lustre: server umount lustre-MDT0000 complete [12649.749137] LustreError: 505547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788342099 with bad export cookie 5145335143668399490 [12649.753512] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12649.754844] LustreError: 505547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12650.012842] Lustre: server umount lustre-MDT0001 complete [12655.873522] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [12657.355306] Key type lgssc unregistered [12657.549669] LNet: 508669:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12657.562870] LNetError: 508669:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12657.580522] LNet: Removed LNI 192.168.203.160@tcp [12658.117299] Key type .llcrypt unregistered [12658.119098] Key type ._llcrypt unregistered [12669.078649] Key type ._llcrypt registered [12669.080011] Key type .llcrypt registered [12669.125501] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [12669.536252] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12669.551362] alg: No test for adler32 (adler32-zlib) [12670.434736] Lustre: Lustre: Build Version: 2.17.57_104_g6cc0f2c [12670.575924] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [12672.183159] Key type lgssc registered [12672.725991] Lustre: Echo OBD driver; http://www.lustre.org/ [12676.496734] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12676.505914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12677.728961] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12679.631810] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [12683.481116] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12683.659605] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12685.602856] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [12688.978854] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12691.626505] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12695.039453] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12695.232479] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12698.391425] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [12701.896124] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12712.761903] loop0: detected capacity change from 0 to 204800000 [12712.839990] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12712.850234] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12712.859459] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12715.450796] loop0: detected capacity change from 0 to 204800000 [12715.534374] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12715.540797] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12715.548384] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12718.040582] loop0: detected capacity change from 0 to 204800000 [12720.850600] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12720.922687] Lustre: fs15246-OST0000: new disk, initializing [12720.925695] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [12720.930494] Lustre: 513504:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12720.963425] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12723.992261] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [12729.157441] loop1: detected capacity change from 0 to 204800000 [12732.146796] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12732.215032] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [12732.256394] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [12732.308247] Lustre: fs15246-MDT0000: new disk, initializing [12732.394542] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12732.408436] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12732.450482] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [12733.473416] Lustre: 514624:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000afeba770 x1875212626855808/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788342199 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [12734.605296] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [12742.566648] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12742.574843] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12742.591427] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12749.798981] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12749.806819] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [12751.211583] Lustre: server umount fs15246-MDT0000 complete [12754.913688] LustreError: 509906:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12754.920816] LustreError: 509906:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12755.962763] Lustre: server umount fs15246-OST0000 complete [12764.130464] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12764.138481] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12764.155339] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12765.156859] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12765.160394] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12765.162198] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12765.168349] Lustre: Skipped 1 previous similar message [12770.278941] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12770.283206] Lustre: Skipped 1 previous similar message [12775.393573] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12775.397059] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12775.404128] Lustre: Skipped 1 previous similar message [12775.502231] Lustre: server umount lustre-OST0000 complete [12780.514163] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12780.524735] Lustre: Skipped 1 previous similar message [12783.603221] Lustre: server umount lustre-MDT0000 complete [12785.478256] LustreError: 509891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788342235 with bad export cookie 412475291597674133 [12785.484522] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12785.490599] LustreError: 509891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12785.689246] Lustre: server umount lustre-MDT0001 complete [12790.237964] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [12791.589900] Key type lgssc unregistered [12791.757301] LNet: 517715:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12791.764078] LNetError: 517715:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12791.777661] LNet: Removed LNI 192.168.203.160@tcp [12792.261775] Key type .llcrypt unregistered [12792.263680] Key type ._llcrypt unregistered [12801.449316] Key type ._llcrypt registered [12801.450877] Key type .llcrypt registered [12801.502130] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [12801.973286] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12802.047163] alg: No test for adler32 (adler32-zlib) [12802.994825] Lustre: Lustre: Build Version: 2.17.57_104_g6cc0f2c [12803.122270] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [12804.711128] Key type lgssc registered [12805.289925] Lustre: Echo OBD driver; http://www.lustre.org/ [12825.457565] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_hostid [12830.565754] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [12850.189596] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12850.201315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12851.347481] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12851.372615] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12851.425314] Lustre: lustre-MDT0000: new disk, initializing [12851.462196] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12851.472540] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12853.403149] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [12859.167650] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12859.239731] Lustre: 522150:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12859.273988] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12859.276778] Lustre: Skipped 1 previous similar message [12859.325668] Lustre: lustre-MDT0001: new disk, initializing [12859.379173] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12859.402474] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12859.410543] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12861.531521] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [12866.137980] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12868.506790] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12871.573409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12871.705232] Lustre: lustre-OST0000: new disk, initializing [12871.707684] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12871.711293] Lustre: 524119:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12871.755985] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12874.458202] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [12879.211598] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12880.914332] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12880.921244] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12880.983431] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12881.344197] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12881.470582] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12883.362108] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12883.457444] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12885.988637] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12885.990153] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12885.995654] LustreError: Skipped 1 previous similar message [12885.996446] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12886.005377] Lustre: Skipped 1 previous similar message [12890.478975] Lustre: server umount lustre-OST0000 complete [12895.200243] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12895.207753] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12895.218626] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12895.221514] Lustre: Skipped 1 previous similar message [12896.225141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12896.225580] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12898.565394] Lustre: server umount lustre-MDT0000 complete [12900.207851] LustreError: 522143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788342350 with bad export cookie 17253620587087627660 [12900.211269] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12900.214437] LustreError: 522143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12900.387072] Lustre: server umount lustre-MDT0001 complete [12911.595545] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_hostid [12916.229839] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [12930.640316] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12621 sec ========= 05:46:20 (1788342380) [12931.481380] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 05:46:21 (1788342381) === [12933.144725] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 05:46:23 (1788342383) === [12950.709162] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [12951.914474] Key type lgssc unregistered [12952.047605] LNet: 530750:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12952.051821] LNetError: 530750:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12952.062874] LNet: Removed LNI 192.168.203.160@tcp [12952.425833] Key type .llcrypt unregistered [12952.427466] Key type ._llcrypt unregistered