[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 480484797 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2480MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002303] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004015] kvm-guest: setup PV IPIs [ 0.007392] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.010139] LSM: Security Framework initializing [ 0.011055] Yama: becoming mindful. [ 0.012040] SELinux: Initializing. [ 0.014030] *** VALIDATE selinux *** [ 0.021670] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026328] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027162] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028116] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029110] *** VALIDATE tmpfs *** [ 0.030447] *** VALIDATE proc *** [ 0.032087] *** VALIDATE cgroup *** [ 0.033009] *** VALIDATE cgroup2 *** [ 0.034271] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035157] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037026] Spectre V2 : User space: Vulnerable [ 0.038010] Speculative Store Bypass: Vulnerable [ 0.041481] debug: unmapping init [mem 0xffffffffb2a59000-0xffffffffb2a60fff] [ 0.043181] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044736] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045026] ... version: 2 [ 0.046018] ... bit width: 48 [ 0.047015] ... generic registers: 4 [ 0.048016] ... value mask: 0000ffffffffffff [ 0.049016] ... max period: 00007fffffffffff [ 0.050017] ... fixed-purpose events: 3 [ 0.051016] ... event mask: 000000070000000f [ 0.053251] rcu: Hierarchical SRCU implementation. [ 0.055448] smp: Bringing up secondary CPUs ... [ 0.056552] x86: Booting SMP configuration: [ 0.057026] .... node #0, CPUs: #1 #2 #3 [ 0.072186] smp: Brought up 1 node, 4 CPUs [ 0.074015] smpboot: Max logical packages: 1 [ 0.075013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.098030] node 0 deferred pages initialised in 21ms [ 0.100492] devtmpfs: initialized [ 0.102286] x86/mm: Memory block size: 128MB [ 0.105972] gcov: version magic: 0x41383552 [ 0.108275] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.111067] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.114574] pinctrl core: initialized pinctrl subsystem [ 0.117316] [ 0.119020] ************************************************************* [ 0.122025] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.124021] ** ** [ 0.128024] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.130025] ** ** [ 0.133023] ** This means that this kernel is built to expose internal ** [ 0.135018] ** IOMMU data structures, which may compromise security on ** [ 0.137021] ** your system. ** [ 0.140026] ** ** [ 0.142030] ** If you see this message and you are not debugging the ** [ 0.144016] ** kernel, report this immediately to your vendor! ** [ 0.145014] ** ** [ 0.147014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.148014] ************************************************************* [ 0.150726] NET: Registered protocol family 16 [ 0.152735] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.155125] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.158077] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.162155] cpuidle: using governor menu [ 0.163647] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.166747] PCI: Using configuration type 1 for base access [ 0.168214] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.179077] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.181040] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.184128] cryptd: max_cpu_qlen set to 1000 [ 0.187282] ACPI: Added _OSI(Module Device) [ 0.188016] ACPI: Added _OSI(Processor Device) [ 0.190020] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.191016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.196723] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.204236] ACPI: Interpreter enabled [ 0.205062] ACPI: PM: (supports S0 S3 S4 S5) [ 0.207019] ACPI: Using IOAPIC for interrupt routing [ 0.209127] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.213412] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.223330] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.225050] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.227050] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.229125] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.233363] acpiphp: Slot [2] registered [ 0.234097] acpiphp: Slot [5] registered [ 0.235086] acpiphp: Slot [6] registered [ 0.236059] acpiphp: Slot [7] registered [ 0.237052] acpiphp: Slot [8] registered [ 0.238031] acpiphp: Slot [9] registered [ 0.238997] acpiphp: Slot [10] registered [ 0.240098] acpiphp: Slot [3] registered [ 0.241029] acpiphp: Slot [4] registered [ 0.241999] acpiphp: Slot [11] registered [ 0.243025] acpiphp: Slot [12] registered [ 0.244076] acpiphp: Slot [13] registered [ 0.245094] acpiphp: Slot [14] registered [ 0.246090] acpiphp: Slot [15] registered [ 0.247147] acpiphp: Slot [16] registered [ 0.248107] acpiphp: Slot [17] registered [ 0.250096] acpiphp: Slot [18] registered [ 0.251085] acpiphp: Slot [19] registered [ 0.252080] acpiphp: Slot [20] registered [ 0.253107] acpiphp: Slot [21] registered [ 0.254085] acpiphp: Slot [22] registered [ 0.255088] acpiphp: Slot [23] registered [ 0.256061] acpiphp: Slot [24] registered [ 0.257084] acpiphp: Slot [25] registered [ 0.258081] acpiphp: Slot [26] registered [ 0.259085] acpiphp: Slot [27] registered [ 0.260088] acpiphp: Slot [28] registered [ 0.261075] acpiphp: Slot [29] registered [ 0.262081] acpiphp: Slot [30] registered [ 0.263070] acpiphp: Slot [31] registered [ 0.264055] PCI host bridge to bus 0000:00 [ 0.265035] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.268021] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.269029] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.271027] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.272025] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.274033] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.276184] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.277897] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.281233] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.289021] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.293112] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.295027] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.297031] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.299021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.301630] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.305775] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.306054] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.309725] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.314049] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.327017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.332024] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.340843] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.347023] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.354025] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.370021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.380316] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.387014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.395019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.411020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.423313] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.429025] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.434022] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.451043] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.464294] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.473018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.485019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.505113] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.516474] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.523020] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.530024] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.542023] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.552068] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.561021] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.566015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.577021] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.587363] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.590479] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.593473] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.596043] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.599319] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.604260] iommu: Default domain type: Passthrough [ 0.606631] SCSI subsystem initialized [ 0.608134] ACPI: bus type USB registered [ 0.610137] usbcore: registered new interface driver usbfs [ 0.612093] usbcore: registered new interface driver hub [ 0.614148] usbcore: registered new device driver usb [ 0.616237] pps_core: LinuxPPS API ver. 1 registered [ 0.619014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.622059] PTP clock support registered [ 0.624119] EDAC MC: Ver: 3.0.0 [ 0.627150] PCI: Using ACPI for IRQ routing [ 0.630320] NetLabel: Initializing [ 0.631012] NetLabel: domain hash size = 128 [ 0.633012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.634091] NetLabel: unlabeled traffic allowed by default [ 0.636242] vgaarb: loaded [ 0.638266] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.640016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.649134] clocksource: Switched to clocksource kvm-clock [ 0.765149] VFS: Disk quotas dquot_6.6.0 [ 0.766556] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.769135] *** VALIDATE ramfs *** [ 0.770308] *** VALIDATE hugetlbfs *** [ 0.771895] pnp: PnP ACPI init [ 0.775163] pnp: PnP ACPI: found 6 devices [ 0.799336] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.803336] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.805710] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.807946] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.810366] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.813248] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.816671] NET: Registered protocol family 2 [ 0.819857] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.825702] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.828948] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.833710] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.836925] TCP: Hash tables configured (established 65536 bind 65536) [ 0.840035] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.842894] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.845588] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.849956] NET: Registered protocol family 1 [ 0.852587] RPC: Registered named UNIX socket transport module. [ 0.854709] RPC: Registered udp transport module. [ 0.856553] RPC: Registered tcp transport module. [ 0.858408] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.861053] NET: Registered protocol family 44 [ 0.863370] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.865854] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.867946] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.870315] PCI: CLS 0 bytes, default 64 [ 0.871514] Unpacking initramfs... [ 2.303110] debug: unmapping init [mem 0xffffa0267cc54000-0xffffa0267ffbffff] [ 2.307384] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.309967] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 2.313321] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.843710] Initialise system trusted keyrings [ 2.845638] Key type blacklist registered [ 2.847660] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.856501] zbud: loaded [ 2.859744] *** VALIDATE nfs *** [ 2.861127] *** VALIDATE nfs4 *** [ 2.862864] pstore: using deflate compression [ 2.866656] Platform Keyring initialized [ 2.991360] NET: Registered protocol family 38 [ 2.993446] Key type asymmetric registered [ 2.995357] Asymmetric key parser 'x509' registered [ 2.997408] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.000869] io scheduler mq-deadline registered [ 3.002667] io scheduler kyber registered [ 3.004079] io scheduler bfq registered [ 3.005884] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.010287] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.012855] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.015424] ACPI: Power Button [PWRF] [ 3.021334] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.027486] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.040792] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.051139] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.079476] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.109592] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.139611] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.144777] Non-volatile memory driver v1.3 [ 3.146377] Linux agpgart interface v0.103 [ 3.180403] virtio_blk virtio1: [vda] 145904 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.183578] vda: detected capacity change from 0 to 74702848 [ 3.203692] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.207312] vdb: detected capacity change from 0 to 1073741824 [ 3.222505] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.225414] vdc: detected capacity change from 0 to 2621440000 [ 3.239360] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.242455] vdd: detected capacity change from 0 to 2621440000 [ 3.257672] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.260951] vde: detected capacity change from 0 to 4294967296 [ 3.279311] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.282494] vdf: detected capacity change from 0 to 4294967296 [ 3.289987] libphy: Fixed MDIO Bus: probed [ 3.295968] usbcore: registered new interface driver usbserial_generic [ 3.297910] usbserial: USB Serial support registered for generic [ 3.299594] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.303415] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.304962] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.306776] mousedev: PS/2 mouse device common for all mice [ 3.309201] rtc_cmos 00:05: RTC can wake from S4 [ 3.311548] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.313620] rtc_cmos 00:05: registered as rtc0 [ 3.317329] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.320458] intel_pstate: CPU model not supported [ 3.324102] hid: raw HID events driver (C) Jiri Kosina [ 3.324346] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.326618] usbcore: registered new interface driver usbhid [ 3.331880] usbhid: USB HID core driver [ 3.333780] drop_monitor: Initializing network drop monitor service [ 3.333911] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.336261] Initializing XFRM netlink socket [ 3.341531] NET: Registered protocol family 10 [ 3.344745] Segment Routing with IPv6 [ 3.346514] NET: Registered protocol family 17 [ 3.348926] mpls_gso: MPLS GSO support [ 3.353906] RAS: Correctable Errors collector initialized. [ 3.356071] AVX version of gcm_enc/dec engaged. [ 3.358037] AES CTR mode by8 optimization enabled [ 3.438077] sched_clock: Marking stable (3438026013, 0)->(4383499208, -945473195) [ 3.441133] registered taskstats version 1 [ 3.442725] Loading compiled-in X.509 certificates [ 3.444470] zswap: loaded using pool lzo/zbud [ 3.471391] Key type big_key registered [ 3.486374] Key type encrypted registered [ 3.488092] ima: No TPM chip found, activating TPM-bypass! [ 3.490246] ima: Allocated hash algorithm: sha1 [ 3.492042] ima: No architecture policies found [ 3.493679] evm: Initialising EVM extended attributes: [ 3.495551] evm: security.selinux [ 3.496863] evm: security.ima [ 3.497936] evm: security.capability [ 3.499199] evm: HMAC attrs: 0x1 [ 3.501680] rtc_cmos 00:05: setting system clock to 2026-08-15 16:39:49 UTC (1786811989) [ 3.507212] debug: unmapping init [mem 0xffffffffb3a03000-0xffffffffb3bfffff] [ 3.509841] debug: unmapping init [mem 0xffffffffb2782000-0xffffffffb2a58fff] [ 3.518071] Write protecting the kernel read-only data: 28672k [ 3.522671] debug: unmapping init [mem 0xffffffffb0e03000-0xffffffffb0ffffff] [ 3.524978] debug: unmapping init [mem 0xffffffffb1714000-0xffffffffb17fffff] [ 3.555321] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.563090] systemd[1]: Detected virtualization kvm. [ 3.565141] systemd[1]: Detected architecture x86-64. [ 3.566611] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.592205] systemd[1]: No hostname configured. [ 3.593696] systemd[1]: Set hostname to . [ 3.595655] random: systemd: uninitialized urandom read (16 bytes read) [ 3.597607] systemd[1]: Initializing machine ID from random generator. [ 3.640108] random: ln: uninitialized urandom read (6 bytes read) [ 3.722042] random: systemd: uninitialized urandom read (16 bytes read) [ 3.725599] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.732355] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.737473] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Reached target Swap. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... Starting Journal Service... [ OK ] Reached target Slices. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Paths. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.412878] device-mapper: uevent: version 1.0.3 [ 4.415221] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.196912] random: fast init done [ 5.200390] virtio_net virtio0 ens2: renamed from eth0 [ 5.256467] scsi host0: ata_piix [ 5.290942] scsi host1: ata_piix [ 5.293219] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.296575] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.874939] dracut-initqueue[582]: RTNETLINK answers: File exists [ 10.003800] random: crng init done [ 10.005312] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.521524] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.798887] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.048121] SELinux: Disabled at runtime. [ 12.103605] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.110427] systemd[1]: Detected virtualization kvm. [ 12.112347] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.558567] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.562739] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.567750] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.571920] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.576491] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.586931] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.597473] systemd[1]: Mounting POSIX Message Queue File System... Mounting POSIX Message Queue File System... [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. Starting Remount Root and Kernel File Systems... [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting Huge Pages File System... Mounting Kernel Debug File System... Starting Apply Kernel Variables... [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target Slices. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of requi[ 12.955177] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS red sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 13.754171] hrtimer: interrupt took 2588180 ns [ 13.803809] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 15.017307] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 15.117750] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 15.535480] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 15.640589] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (7s / no limit)[ 20.349204] Key type dns_resolver registered [*** ] A start job is running for Configur…-only root support (8s / no limit)[ 21.029708] NFS: Registering the id_resolver key type [ 21.032173] Key type id_resolver registered [ 21.033939] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Daily Cleanup of Temporary Directories. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Login Service. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg113-server login: [ 91.331792] libcfs: loading out-of-tree module taints kernel. [ 91.386882] Key type ._llcrypt registered [ 91.390834] Key type .llcrypt registered [ 91.483438] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [ 114.057053] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 117.236248] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 117.290610] alg: No test for adler32 (adler32-zlib) [ 119.661612] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [ 121.851951] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 123.865349] Key type lgssc registered [ 125.851429] Lustre: Echo OBD driver; http://www.lustre.org/ [ 147.534222] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 193.679434] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 208.025686] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 208.063205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 209.352181] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 209.388236] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 209.533268] Lustre: lustre-MDT0000: new disk, initializing [ 209.664569] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 209.695580] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 214.139386] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 227.198251] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 227.316523] Lustre: 6504:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 227.346543] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 227.354149] Lustre: Skipped 1 previous similar message [ 227.514188] Lustre: lustre-MDT0001: new disk, initializing [ 227.621285] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 227.661939] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 227.667554] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 231.862919] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 236.524934] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 245.592657] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 245.936184] Lustre: lustre-OST0000: new disk, initializing [ 245.944962] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 245.951504] Lustre: 8443:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 246.010217] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 251.995689] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 254.519268] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 254.540468] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 254.645900] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 266.605952] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 266.773693] Lustre: lustre-OST0001: new disk, initializing [ 266.782500] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 266.791043] Lustre: 9518:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 266.862122] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 274.391161] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 276.519746] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 276.529936] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 276.564722] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 285.991897] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 296.600280] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 303.420394] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing check_logdir /tmp/testlogs/ [ 309.272565] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing yml_node [ 314.382057] Lustre: DEBUG MARKER: Client: 2.17.57.3 [ 316.845823] Lustre: DEBUG MARKER: MDS: 2.17.57.3 [ 319.496732] Lustre: DEBUG MARKER: OSS: 2.17.57.3 [ 321.059918] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Aug 15 12:45:06 EDT 2026 [ 338.542788] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 340.462115] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 358.368914] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 358.372270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 358.386330] Lustre: Skipped 3 previous similar messages [ 358.400407] Lustre: Skipped 3 previous similar messages [ 363.489749] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 364.049871] Lustre: server umount lustre-MDT0000 complete [ 368.261181] LustreError: 6495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786812354 with bad export cookie 9392008498596664611 [ 368.283582] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 368.607927] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 368.617076] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 368.621963] Lustre: Skipped 1 previous similar message [ 368.631852] Lustre: Skipped 4 previous similar messages [ 373.731294] LustreError: 6512:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 373.732711] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 373.746605] LustreError: 6512:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 373.763363] Lustre: Skipped 1 previous similar message [ 374.795318] Lustre: server umount lustre-MDT0001 complete [ 378.952462] Lustre: server umount lustre-OST0000 complete [ 383.197702] Lustre: server umount lustre-OST0001 complete [ 389.530812] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [ 400.263303] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 444.320705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 444.647791] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 444.683366] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 444.758731] Lustre: lustre-MDT0000: new disk, initializing [ 444.862451] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 444.882349] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 449.545659] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 460.829614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 460.941309] Lustre: 15072:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 460.995470] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 460.999823] Lustre: Skipped 1 previous similar message [ 461.061582] Lustre: lustre-MDT0001: new disk, initializing [ 461.151292] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 461.194294] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 461.207104] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 465.530482] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 474.492228] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 479.080136] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 485.830688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 486.197359] Lustre: lustre-OST0000: new disk, initializing [ 486.203706] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 486.220303] Lustre: 17040:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 486.330668] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 487.364585] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 487.374887] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 487.478412] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 492.834250] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 501.800832] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 506.393442] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 506.524695] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 511.144689] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 511.314505] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 518.113157] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 518.123815] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 518.139863] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 518.145685] Lustre: Skipped 1 previous similar message [ 519.690355] Lustre: server umount lustre-OST0000 complete [ 527.847654] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 527.853327] LustreError: Skipped 1 previous similar message [ 527.860640] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 527.881299] Lustre: Skipped 1 previous similar message [ 527.888838] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 538.079914] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 538.431770] Lustre: server umount lustre-MDT0000 complete [ 538.598327] LustreError: 15083:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 538.618294] LustreError: 15083:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 542.664856] LustreError: 17039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786812528 with bad export cookie 9392008498596666144 [ 542.675205] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 542.685078] LustreError: 17039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 543.097638] Lustre: server umount lustre-MDT0001 complete [ 548.197502] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 12:48:53 (1786812533) [ 550.521466] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 552.281778] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 12:48:57 (1786812537) [ 558.725652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 559.205256] LustreError: 19451:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 559.265206] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 563.285221] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 564.710634] LustreError: 19452:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 569.828525] LustreError: 19451:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 570.915384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 575.647861] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 582.116945] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 586.817375] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 593.162889] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 598.931263] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 606.796572] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 617.443931] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 617.451826] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 617.461687] Lustre: Skipped 1 previous similar message [ 617.476681] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 617.482580] Lustre: Skipped 4 previous similar messages [ 620.512139] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 623.657180] Lustre: server umount lustre-OST0000 complete [ 627.685792] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 627.693983] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 627.710868] Lustre: Skipped 1 previous similar message [ 633.787793] Lustre: server umount lustre-MDT0000 complete [ 635.874264] LustreError: 20179:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 635.895337] LustreError: 20179:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 638.184410] LustreError: 19433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786812624 with bad export cookie 9392008498596667026 [ 638.187886] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 638.195046] LustreError: 19433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 638.806472] Lustre: server umount lustre-MDT0001 complete [ 652.747432] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 655.526142] Key type lgssc unregistered [ 655.745240] LNet: 22986:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 655.751601] LNetError: 22986:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 655.763888] LNet: Removed LNI 192.168.201.113@tcp [ 656.694152] Key type .llcrypt unregistered [ 656.696208] Key type ._llcrypt unregistered [ 681.543913] Key type ._llcrypt registered [ 681.545632] Key type .llcrypt registered [ 681.680131] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 683.067329] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 683.120575] alg: No test for adler32 (adler32-zlib) [ 684.159647] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [ 684.401301] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 686.111229] Key type lgssc registered [ 687.205660] Lustre: Echo OBD driver; http://www.lustre.org/ [ 694.436531] loop: module loaded [ 694.545954] loop0: detected capacity change from 0 to 204800000 [ 694.756483] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 694.775853] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 695.113137] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 700.776644] loop0: detected capacity change from 0 to 204800000 [ 700.961513] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 700.972189] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 700.990751] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 706.290196] loop0: detected capacity change from 0 to 204800000 [ 712.196604] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 712.227895] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 713.547809] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 713.607423] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 713.714441] Lustre: lustre-MDT0000: new disk, initializing [ 713.828128] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 713.857251] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 718.858395] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 729.412085] loop1: detected capacity change from 0 to 204800000 [ 734.887348] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 735.196479] Lustre: lustre-OST07c6: new disk, initializing [ 735.199979] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 735.204978] Lustre: Skipped 1 previous similar message [ 735.209295] Lustre: 26178:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 735.318792] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 741.467860] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 741.475122] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 741.502107] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 742.307618] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 747.495051] Lustre: 25164:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 747.502685] Lustre: 25164:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 747.510042] Lustre: 25164:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 747.536316] Lustre: 25164:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 747.541563] Lustre: 25164:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 747.549223] Lustre: 25164:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 764.553975] Lustre: 25164:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 764.562743] Lustre: 25164:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 764.571360] Lustre: 25164:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 764.576195] Lustre: 25164:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 764.583371] Lustre: 25164:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 764.589525] Lustre: 25164:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 770.083824] Lustre: 25166:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 770.088847] Lustre: 25166:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 770.092689] Lustre: 25166:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 770.104657] Lustre: 25166:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 770.122261] Lustre: 25166:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 770.138421] Lustre: 25166:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 770.143813] Lustre: 25166:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 770.154604] Lustre: 25166:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 770.164160] Lustre: 25166:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 770.177033] Lustre: 25166:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 770.182593] Lustre: 25166:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 770.189781] Lustre: 25166:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 773.482640] Lustre: 25164:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 773.494114] Lustre: 25164:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 773.516492] Lustre: 25164:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 773.529761] Lustre: 25164:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 773.546761] Lustre: 25164:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 773.556513] Lustre: 25164:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 773.562988] Lustre: 25164:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 773.571397] Lustre: 25164:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 773.576613] Lustre: 25164:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 773.584719] Lustre: 25164:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 773.589909] Lustre: 25164:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 773.598561] Lustre: 25164:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 782.494814] Lustre: 25166:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 782.503269] Lustre: 25166:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 782.508180] Lustre: 25166:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 782.515866] Lustre: 25166:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 782.521706] Lustre: 25166:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 782.528751] Lustre: 25166:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 782.536625] Lustre: 25166:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 782.544165] Lustre: 25166:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 782.550739] Lustre: 25166:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 782.556108] Lustre: 25166:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 782.562661] Lustre: 25166:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 782.567277] Lustre: 25166:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 793.609676] Lustre: 25165:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 793.623255] Lustre: 25165:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 793.628142] Lustre: 25165:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 793.633401] Lustre: 25165:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 793.640069] Lustre: 25165:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 793.647319] Lustre: 25165:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 793.661468] Lustre: 25165:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 793.679083] Lustre: 25165:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 793.697152] Lustre: 25165:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 793.710910] Lustre: 25165:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 793.728573] Lustre: 25165:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 793.760902] Lustre: 25165:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 843.765947] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 843.773848] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 843.776801] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 847.899887] Lustre: server umount lustre-OST07c6 complete [ 859.270851] Lustre: server umount lustre-MDT0000 complete [ 876.649930] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 879.339447] Key type lgssc unregistered [ 879.658762] LNet: 28736:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 879.662947] LNetError: 28736:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 879.678826] LNet: Removed LNI 192.168.201.113@tcp [ 880.467310] Key type .llcrypt unregistered [ 880.469713] Key type ._llcrypt unregistered [ 899.861352] Key type ._llcrypt registered [ 899.870361] Key type .llcrypt registered [ 900.036471] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 901.461266] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 901.487240] alg: No test for adler32 (adler32-zlib) [ 902.621336] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [ 902.946595] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 904.612711] Key type lgssc registered [ 905.559782] Lustre: Echo OBD driver; http://www.lustre.org/ [ 912.436571] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 912.471395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 913.918848] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 917.836932] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 923.111539] LustreError: 30024:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 923.131940] LustreError: 30024:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 925.962429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 926.352081] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 931.708357] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 940.448191] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 948.145116] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 956.620211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 957.002741] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 958.118335] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 964.543687] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 973.447462] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 989.156793] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 989.161118] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 989.168760] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 990.176648] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 990.180859] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 993.840812] Lustre: server umount lustre-OST0000 complete [ 997.856720] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 997.867084] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 997.882765] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 997.887049] Lustre: Skipped 1 previous similar message [ 1000.418670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1000.420333] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1005.537736] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1005.545393] Lustre: Skipped 2 previous similar messages [ 1012.191141] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1012.408482] Lustre: server umount lustre-MDT0000 complete [ 1015.776918] LustreError: 30019:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1015.793722] LustreError: 30019:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1015.891705] LustreError: 30004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786813001 with bad export cookie 3890268768307929634 [ 1015.893935] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1015.899951] LustreError: 30004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1016.176201] Lustre: server umount lustre-MDT0001 complete [ 1027.251141] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 1030.231722] Key type lgssc unregistered [ 1030.656572] LNet: 33770:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1030.672494] LNetError: 33770:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1030.690266] LNet: Removed LNI 192.168.201.113@tcp [ 1031.579254] Key type .llcrypt unregistered [ 1031.581474] Key type ._llcrypt unregistered [ 1050.848864] Key type ._llcrypt registered [ 1050.852156] Key type .llcrypt registered [ 1050.949801] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 1052.283212] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1052.298410] alg: No test for adler32 (adler32-zlib) [ 1053.368402] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [ 1053.573813] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 1055.304695] Key type lgssc registered [ 1056.593390] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1065.006977] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1065.057653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1066.516714] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1071.904943] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1076.711680] LustreError: 35039:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1076.748171] LustreError: 35039:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1081.012046] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1081.518469] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1086.503991] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1094.994177] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1100.076273] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1107.052650] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1107.467969] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1111.614987] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1112.872030] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1121.050949] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1137.120137] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1137.135285] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1137.141676] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1137.148812] Lustre: Skipped 1 previous similar message [ 1137.166861] Lustre: Skipped 1 previous similar message [ 1142.204895] Lustre: server umount lustre-OST0000 complete [ 1147.389765] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1147.401503] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1147.407762] Lustre: Skipped 1 previous similar message [ 1147.433920] Lustre: Skipped 1 previous similar message [ 1152.487623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1152.500156] Lustre: Skipped 1 previous similar message [ 1153.744263] Lustre: server umount lustre-MDT0000 complete [ 1157.607848] LustreError: 35040:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1157.645163] LustreError: 35040:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1159.374922] LustreError: 35019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786813145 with bad export cookie 8801891465642350716 [ 1159.389481] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1159.856476] Lustre: server umount lustre-MDT0001 complete [ 1172.113762] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 1175.193982] Key type lgssc unregistered [ 1175.441715] LNet: 38599:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1175.453300] LNetError: 38599:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1175.476713] LNet: Removed LNI 192.168.201.113@tcp [ 1176.208986] Key type .llcrypt unregistered [ 1176.214219] Key type ._llcrypt unregistered [ 1205.196878] Key type ._llcrypt registered [ 1205.199277] Key type .llcrypt registered [ 1205.373886] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 1206.514561] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1206.528845] alg: No test for adler32 (adler32-zlib) [ 1207.560890] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [ 1207.861175] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 1209.575126] Key type lgssc registered [ 1210.929552] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1218.486373] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1218.503855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1219.755654] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1224.208110] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1229.283300] LustreError: 40014:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1229.305396] LustreError: 40014:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1232.939181] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1233.287688] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1238.141778] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1246.619581] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1252.793864] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1260.783489] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1261.197102] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1264.350807] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1268.599428] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1277.533454] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1283.699677] Lustre: server umount lustre-MDT0000 complete [ 1284.076258] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1284.098896] LustreError: 40009:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1284.115173] Lustre: Skipped 2 previous similar messages [ 1284.152197] LustreError: 40009:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1287.536869] LustreError: 39996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786813273 with bad export cookie 1863095803697034576 [ 1287.560229] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1287.571428] LustreError: 39996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1289.184500] LustreError: 40010:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1289.185223] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1289.212834] LustreError: 40010:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1289.257620] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1292.192258] LustreError: 41552:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1292.194475] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1292.212158] LustreError: 41552:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1292.228553] Lustre: Skipped 1 previous similar message [ 1294.320277] Lustre: server umount lustre-MDT0001 complete [ 1316.360388] Lustre: server umount lustre-OST0000 complete [ 1333.702560] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 1336.861634] Key type lgssc unregistered [ 1337.267233] LNet: 43891:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1337.280585] LNetError: 43891:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1338.342878] LNet: Removed LNI 192.168.201.113@tcp [ 1339.484074] Key type .llcrypt unregistered [ 1339.486331] Key type ._llcrypt unregistered [ 1365.669772] Key type ._llcrypt registered [ 1365.672267] Key type .llcrypt registered [ 1365.783372] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 1367.302390] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1367.494256] alg: No test for adler32 (adler32-zlib) [ 1368.709061] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [ 1369.043694] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 1370.712102] Key type lgssc registered [ 1371.722552] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1381.494591] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1381.548456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1383.193507] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1388.673179] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1393.657635] LustreError: 45309:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1393.685122] LustreError: 45309:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1398.773954] LustreError: 45308:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1399.143140] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1399.770467] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1406.010894] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1414.496246] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1420.870698] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1428.074330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1428.305647] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1429.310374] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1435.271447] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1444.143225] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1450.988237] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1450.992032] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1450.999507] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1451.039434] Lustre: Skipped 1 previous similar message [ 1451.524602] Lustre: lustre-OST0000: Not available for connect from 192.168.201.13@tcp (stopping) [ 1451.536060] Lustre: Skipped 1 previous similar message [ 1456.103661] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1456.113335] Lustre: Skipped 1 previous similar message [ 1456.891852] Lustre: server umount lustre-OST0000 complete [ 1476.582498] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1476.587741] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1476.593914] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1476.593923] Lustre: Skipped 1 previous similar message [ 1476.630820] Lustre: Skipped 1 previous similar message [ 1481.696367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1481.701224] Lustre: Skipped 2 previous similar messages [ 1489.375438] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1489.525461] Lustre: server umount lustre-MDT0000 complete [ 1492.966991] LustreError: 45305:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1492.982540] LustreError: 45305:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1493.217906] LustreError: 45288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786813479 with bad export cookie 4566107146983307114 [ 1493.234504] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1499.764395] Lustre: server umount lustre-MDT0001 complete [ 1512.746815] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 1517.089980] Key type lgssc unregistered [ 1517.524281] LNet: 49021:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1517.533063] LNetError: 49021:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1517.547837] LNet: Removed LNI 192.168.201.113@tcp [ 1518.772751] Key type .llcrypt unregistered [ 1518.777455] Key type ._llcrypt unregistered [ 1547.215779] Key type ._llcrypt registered [ 1547.218273] Key type .llcrypt registered [ 1547.378536] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 1548.722511] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1548.756134] alg: No test for adler32 (adler32-zlib) [ 1549.883631] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [ 1550.136690] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 1551.767565] Key type lgssc registered [ 1553.161610] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1563.242421] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1563.256145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1565.073173] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1571.721752] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1575.405709] LustreError: 50438:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1575.435277] LustreError: 50438:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1580.526644] LustreError: 50439:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1582.244339] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1582.951454] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1587.670446] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1596.715154] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1603.587604] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1612.319251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1612.713360] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1620.483894] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1629.156139] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1635.267249] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1638.545435] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1643.111819] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1655.395292] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-1513h07m19s [ 1657.299975] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1659.874474] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1659.885814] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1659.904569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1660.925489] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.13@tcp (stopping) [ 1660.932610] Lustre: Skipped 2 previous similar messages [ 1661.979835] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.13@tcp (stopping) [ 1661.989202] Lustre: Skipped 3 previous similar messages [ 1665.798496] Lustre: server umount lustre-MDT0000 complete [ 1667.043070] LustreError: 52487:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1670.631475] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1673.187410] LustreError: 50438:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1673.223801] LustreError: 50438:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 1678.105962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1678.244554] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1678.305876] LustreError: 49748:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000bb49d190 x1873609790371328/t0(0) o250->MGC192.168.201.113@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1678.598873] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1678.656993] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1679.676907] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1683.767402] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1688.379482] Lustre: DEBUG MARKER: Wait for df (36844) ... [ 1690.649861] Lustre: DEBUG MARKER: done [ 1698.794190] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1698.806510] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1698.816245] Lustre: Skipped 2 previous similar messages [ 1698.825384] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1702.195800] Lustre: server umount lustre-OST0000 complete [ 1710.564809] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1710.580942] Lustre: Skipped 1 previous similar message [ 1710.590525] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1710.596541] Lustre: Skipped 4 previous similar messages [ 1712.535400] Lustre: server umount lustre-MDT0000 complete [ 1716.248894] LustreError: 50418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786813702 with bad export cookie 3292504244361262420 [ 1716.252106] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1716.261585] LustreError: 50418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1716.567054] Lustre: server umount lustre-MDT0001 complete [ 1728.233970] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 1730.775257] Key type lgssc unregistered [ 1731.064659] LNet: 55367:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1731.069979] LNetError: 55367:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1731.092148] LNet: Removed LNI 192.168.201.113@tcp [ 1731.802579] Key type .llcrypt unregistered [ 1731.804483] Key type ._llcrypt unregistered [ 1774.727414] Key type ._llcrypt registered [ 1774.731479] Key type .llcrypt registered [ 1774.825986] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 1776.149444] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1776.192040] alg: No test for adler32 (adler32-zlib) [ 1777.232788] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [ 1777.672311] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 1779.392917] Key type lgssc registered [ 1780.624322] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1787.965960] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1787.977986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1789.225764] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1789.239084] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1789.427332] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1794.015802] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1802.572158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1802.661915] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1802.704833] Lustre: 57576:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1803.040034] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1807.788678] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1815.639584] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1821.193756] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1827.915892] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1828.121696] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1828.277335] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1833.596807] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1834.085680] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1841.664249] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1848.742965] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-1513h10m34s [ 1850.173480] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1856.568244] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1858.732289] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1858.854989] LustreError: 60105:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1879.055356] Lustre: lustre-MDT0000: Client c087ce66-b5ff-49cb-86df-8bfa0a04c506 (at 192.168.201.13@tcp) reconnecting [ 1898.881931] LustreError: 60105:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1901.125202] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1903.419717] Lustre: DEBUG MARKER: done [ 1910.243354] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1910.260055] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1910.290570] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1910.758778] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1911.799555] Lustre: server umount lustre-OST0000 complete [ 1915.878585] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1915.885521] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1915.894977] Lustre: Skipped 1 previous similar message [ 1915.905705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1915.914905] Lustre: Skipped 2 previous similar messages [ 1916.963095] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1921.820682] Lustre: server umount lustre-MDT0000 complete [ 1922.018652] LustreError: 57588:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1922.043894] LustreError: 57588:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1924.800760] LustreError: 57567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786813910 with bad export cookie 11863465965060821572 [ 1924.803612] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1924.808043] LustreError: 57567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1925.043447] Lustre: server umount lustre-MDT0001 complete [ 1936.587864] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 1939.060838] Key type lgssc unregistered [ 1939.335944] LNet: 61908:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1939.346899] LNetError: 61908:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1939.369445] LNet: Removed LNI 192.168.201.113@tcp [ 1940.158099] Key type .llcrypt unregistered [ 1940.161309] Key type ._llcrypt unregistered [ 1983.922869] Key type ._llcrypt registered [ 1983.926454] Key type .llcrypt registered [ 1984.043939] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 1985.081357] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1985.259281] alg: No test for adler32 (adler32-zlib) [ 1986.331582] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [ 1986.561149] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 1988.223175] Key type lgssc registered [ 1989.306769] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1992.093436] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1992.106984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1993.317523] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1993.334749] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1993.482325] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1994.205949] Lustre: Failing over lustre-MDT0000 [ 1994.493619] Lustre: server umount lustre-MDT0000 complete [ 2001.061779] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 13:13:06 (1786813986) [ 2007.245643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2007.801202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2011.849956] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2019.730273] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2019.804773] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2019.827278] Lustre: 64710:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2020.014359] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2023.981027] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2030.679746] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2035.874911] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2041.936934] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2042.136241] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2042.290398] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2048.432624] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2048.673222] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2055.521569] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2061.782510] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2069.476681] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2069.479854] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2069.484848] Lustre: Skipped 2 previous similar messages [ 2069.494935] Lustre: Skipped 1 previous similar message [ 2074.339802] Lustre: server umount lustre-MDT0000 complete [ 2077.882218] LustreError: 64700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786814063 with bad export cookie 14795255236333362516 [ 2077.900294] LustreError: 64700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2077.901793] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2078.208766] Lustre: server umount lustre-MDT0001 complete [ 2081.201881] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2085.154726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2093.129832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2095.839183] Lustre: 63633:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786814065/real 1786814065] req@00000000b9e6ca17 x1873610247545600/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786814081 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2095.877124] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2098.143343] Lustre: 63631:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786814068/real 1786814068] req@000000006fbabe22 x1873610247545728/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786814084 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2100.255210] Lustre: 63631:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786814070/real 1786814070] req@000000000c11e4a8 x1873610247545856/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786814086 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2102.048951] LustreError: 63629:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000d98f34a7 x1873610247546624/t0(0) o250->MGC192.168.201.113@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2102.274564] LustreError: 66557:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2102.295846] LustreError: 66557:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2102.350136] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2102.409715] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2103.327200] Lustre: 63632:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786814073/real 1786814073] req@00000000895b06af x1873610247546112/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786814089 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2104.359372] LustreError: 66558:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2105.391041] LustreError: 66557:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2105.406671] LustreError: 66557:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2105.926891] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2109.407268] Lustre: 63632:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786814079/real 1786814079] req@000000002a5510fb x1873610247546240/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786814095 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2110.433598] LustreError: 66558:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2110.449170] LustreError: 66558:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2112.594498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2112.888318] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2116.091848] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2118.121531] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2122.377243] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2124.263489] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2126.988385] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2133.473460] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2133.478569] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2133.488882] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2134.518099] LustreError: 66557:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2134.534365] LustreError: 66557:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2134.632975] Lustre: server umount lustre-MDT0000 complete [ 2137.642943] LustreError: 66564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786814123 with bad export cookie 14795255236333364791 [ 2137.649645] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2137.652362] LustreError: 66564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2137.921892] Lustre: server umount lustre-MDT0001 complete [ 2143.262323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2151.399044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2156.001398] Lustre: 63632:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786814125/real 1786814125] req@00000000ca5328a4 x1873610247570688/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786814141 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2156.024862] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2156.033105] Lustre: Skipped 2 previous similar messages [ 2162.208056] LustreError: 63629:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000fb63ed73 x1873610247571840/t0(0) o250->MGC192.168.201.113@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2162.395594] LustreError: 66558:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2162.453180] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2162.506787] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2166.002886] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2173.858191] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2175.342646] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2179.475824] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2188.662549] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2194.589451] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2205.153648] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2205.161046] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2205.177337] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2207.843847] Lustre: server umount lustre-MDT0000 complete [ 2211.298415] LustreError: 71158:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2211.307219] LustreError: 71158:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 2211.374883] LustreError: 66563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786814197 with bad export cookie 14795255236333366170 [ 2211.377985] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2211.386929] LustreError: 66563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2211.678341] Lustre: server umount lustre-MDT0001 complete [ 2217.704874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2221.533567] Lustre: DEBUG MARKER: files compared the same [ 2234.428530] Lustre: server umount lustre-OST0000 complete [ 2252.256849] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 2254.938820] Key type lgssc unregistered [ 2255.193644] LNet: 74436:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2255.200170] LNetError: 74436:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2255.215714] LNet: Removed LNI 192.168.201.113@tcp [ 2256.038478] Key type .llcrypt unregistered [ 2256.040060] Key type ._llcrypt unregistered [ 2279.366224] Key type ._llcrypt registered [ 2279.367762] Key type .llcrypt registered [ 2279.464139] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 2280.530959] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2280.686802] alg: No test for adler32 (adler32-zlib) [ 2281.740247] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [ 2281.966195] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 2283.623275] Key type lgssc registered [ 2284.767777] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2294.277347] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2294.292044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2296.395075] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2301.079323] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug +malloc all [ 2306.528954] LustreError: 75852:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2306.540046] LustreError: 75852:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2310.381979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2310.715408] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2315.978657] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug +malloc all [ 2325.043146] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2331.249365] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2339.825357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2340.149264] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2341.188291] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2346.401642] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug +malloc all [ 2354.257511] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2365.923442] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2365.932926] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2365.946841] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2365.959345] Lustre: Skipped 1 previous similar message [ 2371.042060] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2371.049151] Lustre: Skipped 1 previous similar message [ 2376.164229] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2376.172486] Lustre: Skipped 1 previous similar message [ 2377.184795] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2377.359235] Lustre: server umount lustre-OST0000 complete [ 2381.292568] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2381.295110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2381.310268] Lustre: Skipped 1 previous similar message [ 2381.315161] Lustre: Skipped 1 previous similar message [ 2386.407151] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2386.414982] Lustre: Skipped 2 previous similar messages [ 2395.103125] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2395.626138] Lustre: server umount lustre-MDT0000 complete [ 2396.642788] LustreError: 75852:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2396.663312] LustreError: 75852:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2398.900215] LustreError: 75835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786814384 with bad export cookie 5194553941284766924 [ 2398.901634] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2398.914106] LustreError: 75835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2399.629680] Lustre: server umount lustre-MDT0001 complete [ 2414.902513] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 2418.587459] Key type lgssc unregistered [ 2419.182086] LNet: 79390:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2419.196533] LNetError: 79390:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2420.267241] LNet: Removed LNI 192.168.201.113@tcp [ 2421.865054] Key type .llcrypt unregistered [ 2421.866835] Key type ._llcrypt unregistered [ 2451.907839] Key type ._llcrypt registered [ 2451.910630] Key type .llcrypt registered [ 2452.090812] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 2453.921777] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2454.031111] alg: No test for adler32 (adler32-zlib) [ 2455.636379] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [ 2456.144780] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 2457.959561] Key type lgssc registered [ 2460.444812] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2472.127625] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2472.148315] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2487.775860] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2492.895626] LustreError: 80782:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.113@tcp: failed processing log, type 4: rc = -110 [ 2519.583580] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2528.511278] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2540.367348] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2550.915536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2550.976642] Lustre: *** cfs_fail_loc=706, val=0*** [ 2551.249098] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2551.394442] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2556.128301] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2559.463721] LustreError: 81869:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2559.488611] LustreError: 81869:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 2564.588352] LustreError: 81868:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2564.606414] LustreError: 81868:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2565.096168] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2565.665188] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2570.609805] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2579.374842] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2585.990317] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2591.203825] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2591.215241] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2591.244309] Lustre: Skipped 1 previous similar message [ 2591.250734] Lustre: Skipped 1 previous similar message [ 2594.625048] Lustre: server umount lustre-OST0000 complete [ 2601.441937] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2601.477530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2601.484608] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2601.491753] Lustre: Skipped 1 previous similar message [ 2605.994593] Lustre: server umount lustre-MDT0000 complete [ 2606.572866] LustreError: 81873:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2611.681599] LustreError: 82563:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2611.694659] LustreError: 82563:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2612.354470] LustreError: 80790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786814598 with bad export cookie 12838590595252204666 [ 2612.373176] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2612.377265] LustreError: 80790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2612.840576] Lustre: server umount lustre-MDT0001 complete [ 2623.915092] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 2627.903354] Key type lgssc unregistered [ 2628.195619] LNet: 84404:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2628.213248] LNetError: 84404:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2629.285686] LNet: Removed LNI 192.168.201.113@tcp [ 2630.695698] Key type .llcrypt unregistered [ 2630.698786] Key type ._llcrypt unregistered [ 2662.130854] Key type ._llcrypt registered [ 2662.132442] Key type .llcrypt registered [ 2662.224311] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 2663.298167] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2663.375522] alg: No test for adler32 (adler32-zlib) [ 2664.360707] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [ 2664.491151] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 2666.135555] Key type lgssc registered [ 2667.913905] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2678.516606] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2678.539942] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2684.788466] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2693.686342] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2694.073454] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2698.662136] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2699.246760] LustreError: 86510:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2699.269521] LustreError: 86510:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2704.354122] LustreError: 86514:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2704.373831] LustreError: 86514:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2708.534478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2709.011285] LustreError: 86510:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2709.041399] LustreError: 86510:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2709.074702] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2714.084500] LustreError: 86513:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2714.108081] LustreError: 86513:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2716.419405] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2719.217202] LustreError: 87256:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2719.248862] LustreError: 87256:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2726.458712] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2726.470415] Lustre: MGS: Client 7745938d-1b4e-44be-b36e-bf61e44dffb8 (at 0@lo) reconnecting [ 2726.506277] LustreError: 86524:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2726.692053] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2726.824960] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2732.099512] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2748.899175] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2748.926315] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2748.945086] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2752.485893] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2752.492716] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2752.516040] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2754.558974] Lustre: server umount lustre-OST0000 complete [ 2762.721826] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2762.723994] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2762.753740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2762.775054] Lustre: Skipped 2 previous similar messages [ 2765.163045] Lustre: server umount lustre-MDT0000 complete [ 2767.847083] LustreError: 88056:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2767.865180] LustreError: 88056:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2770.211497] Lustre: server umount lustre-MDT0001 complete [ 2774.570209] Lustre: server umount MGS complete [ 2787.106183] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 13:26:11 (1786814771) [ 2819.847067] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [ 2832.194088] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 2871.709762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2876.649357] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2888.233841] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2888.554251] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2888.609951] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2888.682437] Lustre: lustre-MDT0001: new disk, initializing [ 2888.824737] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2888.870439] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2888.883976] Lustre: 93705:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2888.901068] LustreError: 93705:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2893.889837] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2906.057784] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2906.594464] Lustre: lustre-OST0000: new disk, initializing [ 2906.598810] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2906.603321] Lustre: 94472:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2906.682884] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2915.940737] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2917.027918] Lustre: 94826:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@0000000099f862b2 x1873610958481024/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786814919 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2927.933433] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2936.739439] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2936.746360] Lustre: MGS: Client 8001ce37-df40-484c-83f2-d6cf80669922 (at 0@lo) reconnecting [ 2936.812980] Lustre: 92971:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2936.836292] LustreError: 93691:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2936.870499] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2936.950738] Lustre: lustre-MDT0000: new disk, initializing [ 2937.183981] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2937.205995] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2937.222233] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2938.273385] Lustre: 95543:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@0000000091a44689 x1873610958488320/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1786814940 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2942.889968] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2943.400844] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2947.454126] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2947.467203] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2947.505554] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 2972.140825] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2972.145793] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2972.148463] Lustre: Skipped 2 previous similar messages [ 2972.162759] Lustre: Skipped 1 previous similar message [ 2976.042512] Lustre: server umount lustre-OST0000 complete [ 2982.370451] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2982.379172] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2982.388904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2982.423282] Lustre: Skipped 1 previous similar message [ 2986.727249] Lustre: server umount lustre-MDT0000 complete [ 2987.499059] LustreError: 93677:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2987.536142] LustreError: 93677:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2999.377677] Lustre: server umount lustre-MDT0001 complete [ 3003.546517] Lustre: server umount MGS complete [ 3018.795623] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 13:30:03 (1786815003) [ 3042.130205] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 3045.124843] Key type lgssc unregistered [ 3045.355992] LNet: 98046:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3045.365185] LNetError: 98046:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3045.382841] LNet: Removed LNI 192.168.201.113@tcp [ 3046.212927] Key type .llcrypt unregistered [ 3046.215232] Key type ._llcrypt unregistered [ 3067.442479] Key type ._llcrypt registered [ 3067.444853] Key type .llcrypt registered [ 3067.543028] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 3068.619794] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3068.742384] alg: No test for adler32 (adler32-zlib) [ 3069.803460] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [ 3069.983789] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 3071.633571] Key type lgssc registered [ 3072.983554] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3079.840149] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3079.860569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3081.296392] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3085.396850] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3091.426206] LustreError: 99270:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3091.457435] LustreError: 99270:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3094.886411] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3095.390284] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3100.585742] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3108.591303] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3115.425056] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3123.178913] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3123.485328] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3126.593701] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:33) [ 3129.574729] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3139.569474] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3151.339919] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3151.340737] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3151.360245] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3155.441281] Lustre: server umount lustre-OST0000 complete [ 3161.570794] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3161.575306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3161.588017] Lustre: Skipped 2 previous similar messages [ 3161.605556] Lustre: Skipped 2 previous similar messages [ 3165.646829] Lustre: server umount lustre-MDT0000 complete [ 3166.690912] LustreError: 100258:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3166.718729] LustreError: 100258:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3169.756319] LustreError: 99251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786815155 with bad export cookie 818403746937032211 [ 3169.763537] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3170.101899] Lustre: server umount lustre-MDT0001 complete [ 3185.663108] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 3188.913068] Key type lgssc unregistered [ 3189.149555] LNet: 102808:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3189.155626] LNetError: 102808:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3190.183964] LNet: Removed LNI 192.168.201.113@tcp [ 3191.099423] Key type .llcrypt unregistered [ 3191.101673] Key type ._llcrypt unregistered [ 3210.933684] Key type ._llcrypt registered [ 3210.935845] Key type .llcrypt registered [ 3211.007770] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 3211.887053] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3212.073187] alg: No test for adler32 (adler32-zlib) [ 3213.186995] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [ 3213.455813] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 3215.148863] Key type lgssc registered [ 3216.303100] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3223.082399] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3223.090891] LustreError: 103892:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3223.093928] LustreError: 103895:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3223.109895] LustreError: 103892:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4996 [ 3223.119346] LustreError: 103892:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3223.128155] LustreError: 103892:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3223.139111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3224.794687] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3234.278762] LustreError: 103931:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3234.312358] LustreError: 103931:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3234.478672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3234.845638] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3240.583749] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3247.187448] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing lsmod [ 3250.343606] LustreError: 105089:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3250.392636] LustreError: 105099:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3250.399472] LustreError: 105089:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4949 [ 3250.402926] LustreError: 105089:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3250.412145] LustreError: 105089:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3250.432182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3250.824947] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3255.276580] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3255.294199] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3255.315423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3256.290256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3256.291067] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3256.299544] Lustre: Skipped 1 previous similar message [ 3256.328511] Lustre: Skipped 1 previous similar message [ 3259.428770] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3259.445377] Lustre: Skipped 3 previous similar messages [ 3260.016188] Lustre: server umount lustre-MDT0000 complete [ 3264.485754] LustreError: 103931:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3264.499958] LustreError: 103931:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3264.553752] LustreError: 103910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786815250 with bad export cookie 17187009823934553949 [ 3264.555590] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3264.561550] LustreError: 103910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3265.695457] LustreError: 104362:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000003e91c9d6 x1873611534127872/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3265.962190] Lustre: server umount lustre-MDT0001 complete [ 3280.714958] Lustre: server umount lustre-OST0000 complete [ 3289.084202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3289.507255] LustreError: 106165:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3289.585119] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3293.786778] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3294.700714] LustreError: 106166:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3299.815413] LustreError: 106165:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3304.357404] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3305.108572] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3310.584067] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3320.601669] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3329.056122] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3337.787286] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3338.414688] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3345.401832] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3354.055916] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3359.953010] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:65) [ 3370.471741] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3370.473158] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3370.473835] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3370.498665] Lustre: Skipped 1 previous similar message [ 3373.473145] Lustre: server umount lustre-OST0000 complete [ 3380.710858] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3380.717917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3380.737188] Lustre: Skipped 1 previous similar message [ 3380.745477] Lustre: Skipped 1 previous similar message [ 3383.352247] Lustre: server umount lustre-MDT0000 complete [ 3385.827587] LustreError: 106900:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3385.857501] LustreError: 106900:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3387.591386] LustreError: 106145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786815373 with bad export cookie 17187009823934554663 [ 3387.599712] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3387.606581] LustreError: 106145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3388.252746] Lustre: server umount lustre-MDT0001 complete [ 3400.633897] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 3403.759255] Key type lgssc unregistered [ 3404.042318] LNet: 109725:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3404.049435] LNetError: 109725:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3404.068378] LNet: Removed LNI 192.168.201.113@tcp [ 3404.948463] Key type .llcrypt unregistered [ 3404.949991] Key type ._llcrypt unregistered [ 3435.523455] Key type ._llcrypt registered [ 3435.525800] Key type .llcrypt registered [ 3435.655994] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 3436.945340] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3437.014131] alg: No test for adler32 (adler32-zlib) [ 3438.288824] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [ 3438.733148] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 3440.447183] Key type lgssc registered [ 3441.918295] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3451.063591] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3451.104024] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3452.873843] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3458.458992] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3462.114738] LustreError: 111142:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3462.127040] LustreError: 111142:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3467.235932] LustreError: 111143:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3468.656065] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3469.047184] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3474.304199] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3483.007898] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3489.557622] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3497.227182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3497.724098] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3498.732800] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:67 to 0x240000400:97) [ 3504.522818] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3513.941488] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3521.177137] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3526.083692] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3533.796392] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3533.800441] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3533.818108] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3533.853020] Lustre: Skipped 1 previous similar message [ 3538.028620] Lustre: server umount lustre-OST0000 complete [ 3544.033132] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3544.056512] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3544.065032] Lustre: Skipped 2 previous similar messages [ 3549.152629] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3549.160969] Lustre: Skipped 1 previous similar message [ 3553.258773] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3553.265942] Lustre: Skipped 1 previous similar message [ 3556.319931] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3556.480905] Lustre: server umount lustre-MDT0000 complete [ 3558.370136] LustreError: 111143:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3558.395935] LustreError: 111143:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3560.964115] LustreError: 111121:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786815546 with bad export cookie 11687985050830574529 [ 3560.968203] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3560.975107] LustreError: 111121:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3561.372964] Lustre: server umount lustre-MDT0001 complete [ 3573.818645] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 3576.570386] Key type lgssc unregistered [ 3576.826912] LNet: 114787:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3576.833584] LNetError: 114787:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3576.849171] LNet: Removed LNI 192.168.201.113@tcp [ 3577.599320] Key type .llcrypt unregistered [ 3577.601867] Key type ._llcrypt unregistered [ 3597.030666] Key type ._llcrypt registered [ 3597.034412] Key type .llcrypt registered [ 3597.160457] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 3598.741582] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3598.791964] alg: No test for adler32 (adler32-zlib) [ 3599.782578] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [ 3600.058369] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 3601.751795] Key type lgssc registered [ 3603.507435] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3612.135986] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3612.160653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3613.768051] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3619.247585] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3623.905739] LustreError: 116008:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3623.934788] LustreError: 116008:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3628.994418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3629.029393] LustreError: 116007:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3629.433486] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3635.085092] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3643.188967] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3649.445722] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3657.067337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3657.499182] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3658.565339] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:99 to 0x240000400:129) [ 3664.362357] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3672.309324] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3685.344620] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3685.354726] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3685.361174] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3685.371037] Lustre: Skipped 1 previous similar message [ 3685.383538] Lustre: Skipped 1 previous similar message [ 3687.299169] Lustre: server umount lustre-OST0000 complete [ 3695.586316] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3695.603609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3695.603962] Lustre: Skipped 1 previous similar message [ 3695.628789] Lustre: Skipped 1 previous similar message [ 3697.370853] Lustre: server umount lustre-MDT0000 complete [ 3700.705467] LustreError: 117319:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3700.727868] LustreError: 117319:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3701.690015] LustreError: 116750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786815687 with bad export cookie 13264027253206538781 [ 3701.713707] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3701.720742] LustreError: 116750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3702.047725] Lustre: server umount lustre-MDT0001 complete [ 3716.019494] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 3719.267248] Key type lgssc unregistered [ 3719.666634] LNet: 119573:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3719.676919] LNetError: 119573:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3720.741063] LNet: Removed LNI 192.168.201.113@tcp [ 3721.655665] Key type .llcrypt unregistered [ 3721.658519] Key type ._llcrypt unregistered [ 3753.293278] Key type ._llcrypt registered [ 3753.297500] Key type .llcrypt registered [ 3753.439667] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 3754.643325] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3754.790978] alg: No test for adler32 (adler32-zlib) [ 3755.984805] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [ 3756.322790] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 3758.104547] Key type lgssc registered [ 3759.803446] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3767.977202] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3767.990550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3769.475188] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3774.361475] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3779.054102] LustreError: 120994:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3779.071451] LustreError: 120994:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3784.161461] LustreError: 120995:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3784.219857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3784.620137] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3789.951377] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3797.155073] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3803.584739] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3811.255628] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3811.545149] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3814.644767] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:131 to 0x240000400:161) [ 3818.268983] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3826.884935] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3837.779534] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3842.293160] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3842.298268] Lustre: Skipped 2 previous similar messages [ 3846.251168] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3846.314535] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3848.256241] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3848.271017] Lustre: Skipped 2 previous similar messages [ 3855.514099] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3855.522248] Lustre: Skipped 1 previous similar message [ 3860.289134] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.13@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3860.312461] Lustre: Skipped 2 previous similar messages [ 3861.682296] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.13@tcp 192.168.0.[2,10]@tcp [ 3870.279518] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.13/24@tcp in log lustre-MDT0000 [ 3870.287888] Lustre: Skipped 2 previous similar messages [ 3871.589974] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.13/24@tcp [ 3871.595436] Lustre: Skipped 1 previous similar message [ 3882.982012] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3882.983456] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3883.003814] LustreError: Skipped 1 previous similar message [ 3883.039565] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3884.117427] Lustre: server umount lustre-OST0000 complete [ 3891.685645] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3891.691982] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3891.712537] Lustre: Skipped 1 previous similar message [ 3894.650655] Lustre: server umount lustre-MDT0000 complete [ 3896.801394] LustreError: 123936:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3896.825896] LustreError: 123936:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3898.791273] LustreError: 123623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786815884 with bad export cookie 2621330773440269040 [ 3898.794187] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3898.816528] LustreError: 123623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3908.254584] Lustre: server umount lustre-MDT0001 complete [ 3919.302331] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 3921.934760] Key type lgssc unregistered [ 3922.191209] LNet: 125801:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3922.197045] LNetError: 125801:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3923.245912] LNet: Removed LNI 192.168.201.113@tcp [ 3924.054979] Key type .llcrypt unregistered [ 3924.062044] Key type ._llcrypt unregistered [ 3955.367253] Key type ._llcrypt registered [ 3955.371634] Key type .llcrypt registered [ 3955.483597] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 3956.482422] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3956.705201] alg: No test for adler32 (adler32-zlib) [ 3957.732705] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [ 3958.019871] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 3959.695188] Key type lgssc registered [ 3960.704766] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3968.310959] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3968.341536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3969.789830] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3969.834529] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.13/24@tcp [ 3969.842437] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3974.156569] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3979.241821] LustreError: 127364:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3979.261148] LustreError: 127364:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3982.960906] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3983.369168] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3983.394654] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.13/24@tcp [ 3983.492602] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3988.318082] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3996.716686] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4004.794813] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4013.621023] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4013.979941] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4015.105472] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:169 to 0x240000400:193) [ 4020.555763] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4030.470367] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4040.678920] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4040.691079] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4040.718215] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4043.746942] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4043.763038] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4043.770631] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4046.691535] Lustre: server umount lustre-OST0000 complete [ 4053.987202] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4054.007642] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4054.019439] Lustre: Skipped 2 previous similar messages [ 4057.764422] Lustre: server umount lustre-MDT0000 complete [ 4059.112805] LustreError: 127364:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4059.137993] LustreError: 127364:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4062.370827] LustreError: 127345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786816048 with bad export cookie 1615460368189572362 [ 4062.381222] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4063.008677] Lustre: server umount lustre-MDT0001 complete [ 4076.187326] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 4079.168688] Key type lgssc unregistered [ 4079.481879] LNet: 130956:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4079.490128] LNetError: 130956:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4079.527950] LNet: Removed LNI 192.168.201.113@tcp [ 4080.620962] Key type .llcrypt unregistered [ 4080.622355] Key type ._llcrypt unregistered [ 4107.833655] Key type ._llcrypt registered [ 4107.835637] Key type .llcrypt registered [ 4108.011648] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [ 4123.282161] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 4124.585099] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4124.692611] alg: No test for adler32 (adler32-zlib) [ 4125.723072] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [ 4125.968751] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 4127.679757] Key type lgssc registered [ 4128.695877] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4165.510477] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4165.529431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4166.836774] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4166.868093] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4166.935213] Lustre: lustre-MDT0000: new disk, initializing [ 4167.019281] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4167.047878] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4172.008578] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4184.534324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4184.691811] Lustre: 134372:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4184.741198] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4184.745657] Lustre: Skipped 1 previous similar message [ 4184.825413] Lustre: lustre-MDT0001: new disk, initializing [ 4184.932729] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4184.964943] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4184.971904] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4189.468145] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4199.699594] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4206.434283] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4215.446705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4215.783553] Lustre: lustre-OST0000: new disk, initializing [ 4215.791475] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4215.807548] Lustre: 136342:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4215.933995] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4223.201733] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4225.609559] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4225.626441] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4225.730374] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4233.244779] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4237.894451] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4238.089648] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4242.666343] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4242.824371] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4246.505767] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4246.508416] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4246.513412] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4246.517159] LustreError: Skipped 1 previous similar message [ 4246.554254] Lustre: Skipped 1 previous similar message [ 4251.620591] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4251.649346] Lustre: Skipped 2 previous similar messages [ 4256.742435] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4256.752204] Lustre: Skipped 1 previous similar message [ 4259.295613] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4259.689760] Lustre: server umount lustre-OST0000 complete [ 4266.986894] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4266.988111] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4266.998612] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4267.017768] Lustre: Skipped 1 previous similar message [ 4269.992469] Lustre: server umount lustre-MDT0000 complete [ 4273.663596] LustreError: 136341:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786816259 with bad export cookie 9298495883534204587 [ 4273.677193] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4273.905659] Lustre: server umount lustre-MDT0001 complete [ 4282.943502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4283.640508] LustreError: 138561:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4283.671082] LustreError: 138561:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4283.746220] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4288.985624] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4288.999215] LustreError: 138562:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4293.115251] LustreError: 138561:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4298.209335] LustreError: 138562:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4299.418751] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4299.935255] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4306.227883] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4314.644980] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4320.564948] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4327.204452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4327.528947] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4333.369388] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4340.296340] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4344.597776] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4344.842875] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4349.518073] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4349.746301] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4357.571754] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4357.712139] Lustre: lustre-OST0001: new disk, initializing [ 4357.715497] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4357.721038] Lustre: 141932:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4357.808186] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4362.875691] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4366.940255] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4366.947967] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4367.052879] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4370.910836] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4371.165593] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4375.023772] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4375.266427] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4380.717476] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8dcdc42bf000.ost_server_uuid 50 [ 4382.347467] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8dcdc42bf000.ost_server_uuid in FULL state after 0 sec [ 4392.953349] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4392.956252] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4392.963872] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4393.004802] Lustre: Skipped 2 previous similar messages [ 4397.921214] Lustre: server umount lustre-OST0001 complete [ 4398.054586] LustreError: 140385:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4398.073720] LustreError: 140385:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4401.632033] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4401.634150] LustreError: Skipped 1 previous similar message [ 4401.637327] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4401.646589] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4401.650248] Lustre: Skipped 1 previous similar message [ 4407.960341] Lustre: server umount lustre-OST0000 complete [ 4412.394036] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4412.407298] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4412.425843] Lustre: Skipped 1 previous similar message [ 4417.952677] Lustre: server umount lustre-MDT0000 complete [ 4418.537523] LustreError: 138558:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4418.561948] LustreError: 138558:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4421.702763] LustreError: 138542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786816407 with bad export cookie 9298495883534205469 [ 4421.711719] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4421.715050] LustreError: 138542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 4422.271901] Lustre: server umount lustre-MDT0001 complete [ 4442.354908] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 4445.068944] Key type lgssc unregistered [ 4445.548646] LNet: 144938:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4445.557817] LNetError: 144938:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4446.633823] LNet: Removed LNI 192.168.201.113@tcp [ 4447.498965] Key type .llcrypt unregistered [ 4447.501258] Key type ._llcrypt unregistered [ 4491.615502] Key type ._llcrypt registered [ 4491.617976] Key type .llcrypt registered [ 4491.723565] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [ 4508.447499] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 4509.510126] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4509.525391] alg: No test for adler32 (adler32-zlib) [ 4510.769648] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [ 4511.087015] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 4512.752855] Key type lgssc registered [ 4514.129930] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4550.192700] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4550.231352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4551.610812] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4551.702973] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4551.844830] Lustre: lustre-MDT0000: new disk, initializing [ 4552.000931] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4552.030630] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4557.010721] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4567.168598] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4567.238858] Lustre: 149094:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4567.284555] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4567.287931] Lustre: Skipped 1 previous similar message [ 4567.370237] Lustre: lustre-MDT0001: new disk, initializing [ 4567.528551] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4567.568431] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4567.575594] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4572.804272] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4582.980891] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4590.122856] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4597.038520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4597.317833] Lustre: lustre-OST0000: new disk, initializing [ 4597.325152] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4597.339093] Lustre: 151061:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4597.432927] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4604.263658] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4607.038739] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4607.069862] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4607.180769] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4614.138485] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4618.748609] Lustre: Failing over lustre-OST0000 [ 4619.100810] Lustre: server umount lustre-OST0000 complete [ 4635.113055] Lustre: 147317:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786816605/real 1786816605] req@00000000254dbd20 x1873612895192320/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1786816621 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4635.144030] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4638.050085] Lustre: 147318:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786816608/real 1786816608] req@0000000003ee7a30 x1873612895193088/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786816624 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4638.092921] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4639.007604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4639.135721] Lustre: 147316:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786816608/real 1786816608] req@00000000b423c1f0 x1873612895193344/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786816624 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4639.162028] Lustre: 147316:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4639.473354] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4639.490266] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4643.528201] Lustre: 147317:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786816613/real 1786816613] req@00000000660b59cd x1873612895194496/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786816629 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4645.610764] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4646.313656] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4648.718930] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4648.719281] Lustre: lustre-OST0000: Recovery over after 0:03, of 3 clients 3 recovered and 0 were evicted. [ 4649.698534] Lustre: 147319:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786816619/real 1786816619] req@0000000088b9fe3e x1873612895195648/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786816635 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4649.721032] Lustre: 147319:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4651.112913] Lustre: Failing over lustre-MDT0000 [ 4651.391721] Lustre: server umount lustre-MDT0000 complete [ 4654.561294] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4654.572584] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4657.631730] LustreError: 150378:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4657.658105] LustreError: 150378:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 4661.216851] LustreError: 149105:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4661.230828] LustreError: 149105:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4662.750557] LustreError: 149100:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4666.377627] LustreError: 149101:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4666.404229] LustreError: 149101:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4670.775464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4670.888829] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4671.141224] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4671.195203] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4672.990758] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4676.593763] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4676.601934] Lustre: Skipped 1 previous similar message [ 4676.645315] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4676.682687] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4676.705963] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4686.818349] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4686.846399] Lustre: Skipped 2 previous similar messages [ 4686.861798] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4686.871683] Lustre: Skipped 1 previous similar message [ 4691.545907] Lustre: server umount lustre-OST0000 complete [ 4697.064222] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4697.068267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4697.069326] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4697.079563] Lustre: Skipped 2 previous similar messages [ 4701.602849] Lustre: server umount lustre-MDT0000 complete [ 4702.177062] LustreError: 149933:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4702.202418] LustreError: 149933:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4705.595329] LustreError: 151060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786816691 with bad export cookie 6194358771210555956 [ 4705.596580] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4705.618949] LustreError: 151060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4705.900646] Lustre: server umount lustre-MDT0001 complete [ 4719.155792] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 4722.432754] Key type lgssc unregistered [ 4722.825718] LNet: 154718:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4722.831852] LNetError: 154718:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4722.844985] LNet: Removed LNI 192.168.201.113@tcp [ 4723.731413] Key type .llcrypt unregistered [ 4723.736778] Key type ._llcrypt unregistered [ 4762.502762] Key type ._llcrypt registered [ 4762.505795] Key type .llcrypt registered [ 4762.627243] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 4764.362756] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4764.408577] alg: No test for adler32 (adler32-zlib) [ 4765.517444] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [ 4765.965716] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 4767.647172] Key type lgssc registered [ 4768.915705] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4777.380215] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4777.394597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4778.794828] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4783.505781] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4788.194170] LustreError: 156240:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4788.213067] LustreError: 156240:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4793.322575] LustreError: 156241:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4793.567768] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4797.633044] LustreError: 156269:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4797.648439] LustreError: 156269:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4797.657504] LustreError: 156269:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 4798.397102] Lustre: server umount lustre-MDT0000 complete [ 4816.044381] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [ 4826.529376] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 4867.517117] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4867.803760] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4867.841922] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4867.996372] Lustre: lustre-MDT0000: new disk, initializing [ 4868.212285] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4868.234830] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4873.080211] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4883.969454] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4884.118292] Lustre: 159575:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4884.171476] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4884.174540] Lustre: Skipped 1 previous similar message [ 4884.228188] Lustre: lustre-MDT0001: new disk, initializing [ 4884.318416] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4884.351458] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4884.369870] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4889.199127] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4900.047355] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4908.147509] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4915.678304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4915.970357] Lustre: lustre-OST0000: new disk, initializing [ 4915.975354] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4915.983651] Lustre: 161555:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4916.097600] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4921.931897] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4921.942352] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4922.095772] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4923.494327] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4933.867680] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4938.133116] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4940.068144] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 14:02:05 (1786816925) [ 5219.300279] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5219.307986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5219.315632] Lustre: Skipped 2 previous similar messages [ 5219.318457] Lustre: Skipped 1 previous similar message [ 5224.416119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5224.422052] Lustre: Skipped 3 previous similar messages [ 5225.474461] Lustre: server umount lustre-MDT0000 complete [ 5229.536856] LustreError: 159586:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5229.565524] LustreError: 159586:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5229.868052] LustreError: 159567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786817215 with bad export cookie 7901149505464504374 [ 5229.869963] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5229.885572] LustreError: 159567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5234.662114] LustreError: 163335:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5234.662849] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5234.703063] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5239.778167] LustreError: 159581:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5239.778196] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5244.383378] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5244.802745] Lustre: server umount lustre-MDT0001 complete [ 5263.327639] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5263.681923] Lustre: server umount lustre-OST0000 complete [ 5275.881698] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [ 5291.826169] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 5335.815370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5336.076309] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5336.123626] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5336.227851] Lustre: lustre-MDT0000: new disk, initializing [ 5336.330719] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5336.358821] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5341.346418] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 5353.511921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5353.665845] Lustre: 166255:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5353.724667] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5353.727338] Lustre: Skipped 1 previous similar message [ 5353.776343] Lustre: lustre-MDT0001: new disk, initializing [ 5353.846765] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5353.878053] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5353.899154] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5359.611317] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 5369.349527] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5376.544182] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5385.001101] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5385.524180] Lustre: lustre-OST0000: new disk, initializing [ 5385.528586] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5385.537398] Lustre: 168222:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5385.694381] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5392.955394] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 5394.587771] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5394.605184] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5394.721819] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5403.924665] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5409.423244] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5409.722121] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5415.030618] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5415.449952] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5420.519269] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5420.527590] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5420.534100] LustreError: Skipped 1 previous similar message [ 5420.538691] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5420.549564] Lustre: Skipped 1 previous similar message [ 5420.573389] Lustre: Skipped 2 previous similar messages [ 5423.932627] Lustre: server umount lustre-OST0000 complete [ 5430.756336] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5430.779820] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5430.781525] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5430.785319] Lustre: Skipped 1 previous similar message [ 5435.008070] Lustre: server umount lustre-MDT0000 complete [ 5435.877496] LustreError: 166265:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5435.901609] LustreError: 166265:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5439.553956] LustreError: 166245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786817425 with bad export cookie 7901149505464663204 [ 5439.572380] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5440.103415] Lustre: server umount lustre-MDT0001 complete [ 5450.074471] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 14:10:35 (1786817435) [ 5467.317951] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [ 5480.663703] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 5524.711384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5524.982372] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5525.021901] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5525.080345] Lustre: lustre-MDT0000: new disk, initializing [ 5525.211394] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5525.239549] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5530.658335] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 5541.760782] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5541.996682] Lustre: 172979:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5542.426922] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5547.871241] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 5558.908651] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5565.478994] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5572.125725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5572.359244] Lustre: lustre-OST0000: new disk, initializing [ 5572.362903] Lustre: Skipped 1 previous similar message [ 5572.365521] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5572.368138] Lustre: Skipped 2 previous similar messages [ 5572.373076] Lustre: 174959:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5579.408786] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 5581.416289] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5581.425614] Lustre: Skipped 1 previous similar message [ 5581.433434] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5581.541356] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5589.479711] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5597.158176] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5597.168398] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5597.174359] LustreError: Skipped 1 previous similar message [ 5597.177731] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5597.177740] Lustre: Skipped 1 previous similar message [ 5597.207200] Lustre: Skipped 1 previous similar message [ 5602.606357] Lustre: server umount lustre-OST0000 complete [ 5607.396555] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5607.417632] Lustre: Skipped 1 previous similar message [ 5621.216111] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5621.443980] Lustre: server umount lustre-MDT0000 complete [ 5622.758128] LustreError: 172989:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5622.800659] LustreError: 172989:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5626.552406] LustreError: 172973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786817612 with bad export cookie 7901149505464664086 [ 5626.560415] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5626.571187] LustreError: 172973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 5626.849904] Lustre: server umount lustre-MDT0001 complete [ 5636.776713] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 14:13:41 (1786817621) [ 5654.919723] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [ 5667.389248] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 5709.247405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5709.549317] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5709.571444] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5709.640622] Lustre: lustre-MDT0000: new disk, initializing [ 5709.744641] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5709.750437] Lustre: Skipped 2 previous similar messages [ 5709.773386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5714.542156] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 5729.651477] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5729.941475] Lustre: 179283:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5730.390291] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5736.587071] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 5748.868619] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5756.250940] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5764.530731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5764.771220] Lustre: 181261:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5771.894120] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5772.083317] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5772.248790] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 5782.829808] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5792.737083] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5792.750454] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5792.760483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5792.772713] Lustre: Skipped 1 previous similar message [ 5792.787089] Lustre: Skipped 10 previous similar messages [ 5797.333328] Lustre: server umount lustre-OST0000 complete [ 5801.963592] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5801.981192] LustreError: Skipped 1 previous similar message [ 5807.537183] Lustre: server umount lustre-MDT0000 complete [ 5808.103596] LustreError: 180469:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5808.116943] LustreError: 180469:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5811.253866] LustreError: 179275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786817797 with bad export cookie 7901149505464665171 [ 5811.264894] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5811.269749] LustreError: 179275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5811.693581] Lustre: server umount lustre-MDT0001 complete [ 5825.362893] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 5828.691624] Key type lgssc unregistered [ 5829.065479] LNet: 183120:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5829.076251] LNetError: 183120:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5829.096336] LNet: Removed LNI 192.168.201.113@tcp [ 5830.035508] Key type .llcrypt unregistered [ 5830.039350] Key type ._llcrypt unregistered [ 5858.757789] Key type ._llcrypt registered [ 5858.759905] Key type .llcrypt registered [ 5858.853932] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 5860.097810] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5860.176949] alg: No test for adler32 (adler32-zlib) [ 5861.177948] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [ 5861.413079] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 5863.057289] Key type lgssc registered [ 5864.285309] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5872.837951] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5872.865772] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5874.431770] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5879.273504] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 5884.905994] LustreError: 184537:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5884.927786] LustreError: 184537:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5888.493195] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5889.127749] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5894.335533] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 5902.229854] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5908.859595] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5916.549781] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5917.000899] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5919.084200] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5924.036895] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 5932.778845] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5954.023473] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5954.029716] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5954.044421] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5954.060688] Lustre: Skipped 1 previous similar message [ 5956.776853] Lustre: server umount lustre-OST0000 complete [ 5964.258665] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5964.261105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5964.276188] Lustre: Skipped 2 previous similar messages [ 5964.291257] Lustre: Skipped 1 previous similar message [ 5967.511994] Lustre: server umount lustre-MDT0000 complete [ 5969.389052] LustreError: 186524:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5969.419621] LustreError: 186524:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5971.673558] LustreError: 185274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786817957 with bad export cookie 16833274565879160545 [ 5971.677484] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5971.692532] LustreError: 185274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5972.371233] Lustre: server umount lustre-MDT0001 complete [ 5985.994091] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 5989.300445] Key type lgssc unregistered [ 5989.643845] LNet: 188098:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5989.653328] LNetError: 188098:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5989.670484] LNet: Removed LNI 192.168.201.113@tcp [ 5990.450635] Key type .llcrypt unregistered [ 5990.452483] Key type ._llcrypt unregistered [ 6017.880924] Key type ._llcrypt registered [ 6017.885461] Key type .llcrypt registered [ 6018.138424] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 6019.304886] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6019.317083] alg: No test for adler32 (adler32-zlib) [ 6020.593196] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [ 6020.906228] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 6022.599189] Key type lgssc registered [ 6024.109182] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6033.142401] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6033.185698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6034.824970] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6039.971326] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6044.167120] LustreError: 189516:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6044.201518] LustreError: 189516:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6049.249403] LustreError: 189517:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6049.866181] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6050.261498] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6056.400667] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6064.885757] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6071.432613] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6080.117484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6080.674558] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6089.909473] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6100.450122] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6104.319231] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6114.785166] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6114.791694] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6114.806072] Lustre: Skipped 1 previous similar message [ 6114.806148] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6114.820716] Lustre: Skipped 1 previous similar message [ 6117.954682] Lustre: lustre-OST0000: Not available for connect from 192.168.201.13@tcp (stopping) [ 6119.568505] Lustre: server umount lustre-OST0000 complete [ 6129.086716] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8dcdd02c8800.ost_server_uuid 50 [ 6131.359192] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6147.555789] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6147.569729] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6147.601069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6149.603489] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6149.604879] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6152.380554] Lustre: server umount lustre-MDT0000 complete [ 6154.729901] LustreError: 189516:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6154.767440] LustreError: 189516:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6156.786641] LustreError: 189499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786818142 with bad export cookie 375316856506473347 [ 6156.793366] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6156.796286] LustreError: 189499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6157.116920] Lustre: server umount lustre-MDT0001 complete [ 6167.128684] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 14:22:31 (1786818151) [ 6173.792305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6174.127058] LustreError: 193455:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6174.143252] LustreError: 193455:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6174.200338] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6178.870444] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6179.302669] LustreError: 193456:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6186.915167] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6187.318380] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6192.034432] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6200.179549] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6205.887626] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6212.990863] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6213.439799] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6219.608910] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6220.664031] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6229.131787] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6236.993852] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6237.208979] Lustre: lustre-OST0001: new disk, initializing [ 6237.212855] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6237.218514] Lustre: 196298:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6237.297374] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6244.048349] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6245.034336] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6245.048524] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6245.200457] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6253.264618] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6258.657451] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6258.669545] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6258.689708] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6258.706685] Lustre: Skipped 1 previous similar message [ 6260.710169] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6262.263435] Lustre: server umount lustre-OST0000 complete [ 6264.307799] LustreError: 196097:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6264.324969] LustreError: 196097:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6268.786565] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6269.043277] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6273.920104] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6274.085374] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6281.188505] LustreError: 196097:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6281.217736] LustreError: 196097:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 6291.427763] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6291.434461] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6291.440038] LustreError: Skipped 1 previous similar message [ 6291.442890] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6291.442897] Lustre: Skipped 3 previous similar messages [ 6291.469561] Lustre: Skipped 2 previous similar messages [ 6293.968756] Lustre: server umount lustre-OST0001 complete [ 6300.128447] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6300.136698] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6304.532178] Lustre: server umount lustre-MDT0000 complete [ 6308.056606] LustreError: 193437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786818294 with bad export cookie 375316856506474418 [ 6308.061542] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6308.066602] LustreError: 193437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6308.377316] Lustre: server umount lustre-MDT0001 complete [ 6330.405393] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 14:25:15 (1786818315) [ 6336.635851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6336.957499] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6337.009721] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6337.240738] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6342.177870] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6351.649651] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6351.760898] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6351.812940] Lustre: 199736:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6357.224789] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6366.218288] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6373.445634] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6382.875294] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6383.124625] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6389.395705] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6391.003199] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6397.803841] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6405.774516] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6405.925647] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6406.367569] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6406.373534] Lustre: Skipped 2 previous similar messages [ 6413.568643] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6415.602187] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6421.905842] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6426.096498] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6426.099320] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6426.126400] Lustre: Skipped 2 previous similar messages [ 6426.139184] Lustre: Skipped 5 previous similar messages [ 6431.047328] Lustre: server umount lustre-OST0000 complete [ 6431.216525] LustreError: 201590:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6431.260765] LustreError: 201590:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 6451.711958] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6454.562864] Lustre: server umount lustre-OST0001 complete [ 6459.872412] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6459.897466] Lustre: Skipped 2 previous similar messages [ 6459.908158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6459.926529] Lustre: Skipped 3 previous similar messages [ 6464.572908] Lustre: server umount lustre-MDT0000 complete [ 6469.384995] LustreError: 199730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786818455 with bad export cookie 375316856506475818 [ 6469.400512] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6469.408138] LustreError: 199730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6470.165488] Lustre: server umount lustre-MDT0001 complete [ 6499.311760] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 14:28:04 (1786818484) [ 6516.927652] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [ 6528.504125] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 6575.170877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6575.390337] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6575.421139] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6575.536779] Lustre: lustre-MDT0000: new disk, initializing [ 6575.630847] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6575.649429] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6581.721976] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6595.807431] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6595.988405] Lustre: 207831:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6596.120801] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6596.126627] Lustre: Skipped 1 previous similar message [ 6596.254408] Lustre: lustre-MDT0001: new disk, initializing [ 6596.467114] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6596.490882] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6602.566810] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6613.768889] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6621.667798] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6630.865825] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6631.229602] Lustre: lustre-OST0000: new disk, initializing [ 6631.233424] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6631.240969] Lustre: 209805:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6639.193085] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6641.238942] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6641.250835] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6641.487322] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6650.527929] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6656.722673] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6656.955277] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6662.250931] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6662.464801] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6667.238585] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6667.243270] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6667.258153] Lustre: Skipped 1 previous similar message [ 6667.267765] LustreError: Skipped 2 previous similar messages [ 6667.268971] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6667.285515] Lustre: Skipped 3 previous similar messages [ 6670.724433] Lustre: server umount lustre-OST0000 complete [ 6680.941635] Lustre: server umount lustre-MDT0000 complete [ 6682.593203] LustreError: 207839:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6682.620698] LustreError: 207839:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 14 previous similar messages [ 6685.228834] LustreError: 208638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786818671 with bad export cookie 375316856506477610 [ 6685.230577] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6685.241237] LustreError: 208638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6685.872623] Lustre: server umount lustre-MDT0001 complete [ 6695.147521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6700.796122] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6710.483262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6715.462841] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6723.561844] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6731.255856] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6739.410282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6747.204185] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6755.583788] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6762.181484] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6762.409872] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6767.070051] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6767.237911] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6770.662131] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6770.675335] LustreError: Skipped 1 previous similar message [ 6775.313863] Lustre: server umount lustre-OST0000 complete [ 6782.606230] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6787.343653] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6817.064856] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6824.208250] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6832.323749] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6835.836343] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6835.857772] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6835.861503] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6835.880443] Lustre: Skipped 1 previous similar message [ 6835.916248] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 6846.432358] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6846.443285] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6846.456387] Lustre: Skipped 5 previous similar messages [ 6846.465192] Lustre: Skipped 4 previous similar messages [ 6847.934579] Lustre: server umount lustre-OST0000 complete [ 6858.913992] Lustre: server umount lustre-MDT0000 complete [ 6861.795085] LustreError: 212018:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6861.811693] LustreError: 212018:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6863.132600] LustreError: 212004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786818849 with bad export cookie 375316856506478492 [ 6863.149631] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6863.724422] Lustre: server umount lustre-MDT0001 complete [ 6873.811488] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 14:34:18 (1786818858) [ 6881.847382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6882.783089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6882.786253] Lustre: Skipped 6 previous similar messages [ 6888.659368] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6899.776690] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6904.801469] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6914.202489] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6922.603904] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6930.982093] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6940.091824] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6949.078265] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6955.342778] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6955.616470] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6961.773399] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6962.087775] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6969.807488] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6970.007371] Lustre: lustre-OST0001: new disk, initializing [ 6970.011282] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6970.018278] Lustre: 221361:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6977.092612] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6978.178643] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6978.190402] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6978.324959] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6988.120488] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6993.413446] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6993.635113] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6998.628179] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6998.829457] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7015.391167] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7015.525802] Lustre: server umount lustre-OST0001 complete [ 7023.311669] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7023.668626] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7029.439360] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7029.717089] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7060.828714] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7066.599572] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7066.609308] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7066.609628] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7066.632473] Lustre: Skipped 1 previous similar message [ 7068.892275] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 7077.610495] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7079.278647] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7081.955412] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7081.966342] LustreError: Skipped 4 previous similar messages [ 7087.826681] Lustre: server umount lustre-OST0001 complete [ 7106.019252] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7106.039744] Lustre: Skipped 6 previous similar messages [ 7106.050808] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7106.056107] Lustre: Skipped 15 previous similar messages [ 7117.797876] LustreError: 219811:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7117.817373] LustreError: 219811:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 39 previous similar messages [ 7119.327368] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7119.592041] Lustre: server umount lustre-OST0000 complete [ 7130.500042] Lustre: server umount lustre-MDT0000 complete [ 7135.084710] LustreError: 217972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786819121 with bad export cookie 375316856506479458 [ 7135.087588] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7135.126957] LustreError: 217972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 7141.752525] Lustre: server umount lustre-MDT0001 complete [ 7170.245275] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 14:39:15 (1786819155) [ 7179.149622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7179.484828] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7179.517297] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7186.944210] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 7198.450608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7198.517980] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7198.537671] Lustre: 226613:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7204.499511] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 7214.698767] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7223.024881] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7232.942871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7233.495723] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7241.941732] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 7251.923714] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7261.450290] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7261.529505] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7269.077426] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 7278.093042] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7279.945920] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7285.406990] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7285.691330] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7291.359240] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7291.611427] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7298.807438] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8dcdc3dc5800.ost_server_uuid 50 [ 7301.051906] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8dcdc3dc5800.ost_server_uuid in IDLE state after 0 sec [ 7302.113325] Lustre: Permanently deactivating lustre-OST0001 [ 7302.148430] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7306.105581] Lustre: Permanently reactivating lustre-OST0001 [ 7313.782598] Lustre: server umount lustre-OST0001 complete [ 7324.948957] Lustre: server umount lustre-OST0000 complete [ 7336.050184] Lustre: server umount lustre-MDT0000 complete [ 7340.315085] LustreError: 226606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786819326 with bad export cookie 375316856506480746 [ 7340.324194] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7340.338594] LustreError: 226606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7340.684110] Lustre: server umount lustre-MDT0001 complete [ 7370.415772] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 14:42:34 (1786819354) [ 7379.884626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7380.164648] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7380.220634] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7380.237433] Lustre: Skipped 5 previous similar messages [ 7387.016857] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 7396.811046] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7396.976100] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7397.044720] Lustre: 233312:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7397.451477] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7397.466541] Lustre: Skipped 9 previous similar messages [ 7403.267816] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 7414.350984] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7422.629276] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7431.218639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7431.521526] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7431.551713] Lustre: Permanently deactivating lustre-OST0000 [ 7431.575364] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7438.885455] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7439.733600] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 7448.456934] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7457.595778] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7457.746687] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7465.885734] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7466.186722] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 7477.203857] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7479.860956] Lustre: Permanently reactivating lustre-OST0000 [ 7479.900384] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7479.903372] Lustre: Skipped 2 previous similar messages [ 7486.614588] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7486.621905] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 47 seconds. I think it's dead, and I am evicting it. exp 000000004ca7fc75, cur 1786819472 deadline 1786819455 last 1786819425 [ 7486.623394] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7486.654860] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7486.695222] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7496.673280] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7496.676632] LustreError: Skipped 7 previous similar messages [ 7503.103780] Lustre: server umount lustre-OST0001 complete [ 7513.703218] Lustre: server umount lustre-OST0000 complete [ 7524.716095] Lustre: server umount lustre-MDT0000 complete [ 7529.397041] LustreError: 233304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786819515 with bad export cookie 375316856506482426 [ 7529.405579] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7529.411810] LustreError: 233304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7529.785681] Lustre: server umount lustre-MDT0001 complete [ 7543.253823] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 7546.222651] Key type lgssc unregistered [ 7546.459645] LNet: 238198:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7546.470211] LNetError: 238198:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7547.502740] LNet: Removed LNI 192.168.201.113@tcp [ 7548.909729] Key type .llcrypt unregistered [ 7548.917049] Key type ._llcrypt unregistered [ 7577.460399] Key type ._llcrypt registered [ 7577.462376] Key type .llcrypt registered [ 7577.549839] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 7578.962867] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7579.248808] alg: No test for adler32 (adler32-zlib) [ 7580.297833] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [ 7580.543104] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 7582.271434] Key type lgssc registered [ 7583.627276] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7595.437850] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7595.462990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7596.954264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7602.508684] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 7606.251343] LustreError: 239667:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7606.270616] LustreError: 239667:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7611.382714] LustreError: 239668:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7612.391028] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7612.561881] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7612.610458] Lustre: 239656:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7612.623438] Lustre: Permanently deactivating lustre-MDT0001 [ 7612.642865] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7612.966202] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7618.671950] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 7620.087881] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7627.902426] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7635.861478] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7644.030417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7644.443251] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7651.705859] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 7657.778382] LustreError: 241510:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7657.797687] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7660.147137] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7663.075018] LustreError: 241510:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7666.161926] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7667.194034] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7667.404853] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7672.829572] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7672.858212] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7673.999760] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 7682.748224] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7686.232746] Lustre: Permanently reactivating lustre-MDT0001 [ 7686.248258] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7686.250654] Lustre: Skipped 2 previous similar messages [ 7688.759331] Lustre: lustre-MDT0001: Client b700ef01-fc2f-49cf-9028-cbbddb1a6746 (at 192.168.201.13@tcp) reconnecting [ 7692.978344] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7692.997684] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7693.017900] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7693.043496] LustreError: 243316:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 7693.051694] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7693.054527] LustreError: 243316:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 7693.054642] LustreError: 243316:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.113@tcp: cfg command failed: rc = -108 [ 7693.054664] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 7693.054664] [ 7693.055056] LustreError: 239659:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 7696.881370] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7698.885987] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7705.209818] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7705.363236] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7710.318750] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7710.564371] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7713.474713] Lustre: Permanently deactivating lustre-MDT0001 [ 7713.509322] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7713.514619] Lustre: Skipped 2 previous similar messages [ 7723.624067] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7734.242037] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7734.257898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7734.263619] Lustre: Skipped 1 previous similar message [ 7734.282972] Lustre: Skipped 3 previous similar messages [ 7735.908174] Lustre: server umount lustre-MDT0000 complete [ 7739.364995] LustreError: 241122:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7739.382152] LustreError: 241122:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 7739.493081] LustreError: 239647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786819725 with bad export cookie 2347804086723806271 [ 7739.502744] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7739.506560] LustreError: 239647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7739.936726] Lustre: server umount lustre-MDT0001 complete [ 7746.512468] Lustre: server umount lustre-OST0000 complete [ 7760.602234] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 14:49:05 (1786819745) [ 7771.826283] Lustre: server umount lustre-OST0001 complete [ 7779.760343] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [ 7793.073331] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 7830.448784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7830.734973] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7830.744781] Lustre: Skipped 2 previous similar messages [ 7830.762993] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7830.837377] Lustre: lustre-MDT0000: new disk, initializing [ 7830.913927] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7830.934703] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7835.216753] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 7846.523289] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7846.675309] Lustre: 247972:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7846.773507] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7846.777616] Lustre: Skipped 1 previous similar message [ 7846.887941] Lustre: lustre-MDT0001: new disk, initializing [ 7846.990794] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7847.048120] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7847.064454] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7851.383869] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 7862.511795] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7868.161264] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7876.194643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7876.471599] Lustre: lustre-OST0000: new disk, initializing [ 7876.474339] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7876.482312] Lustre: 249946:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7876.561906] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7882.336474] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7882.351057] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7882.524957] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7882.862230] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 7892.741178] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7894.941254] LustreError: 247980:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7904.246703] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7904.526878] Lustre: lustre-OST0001: new disk, initializing [ 7904.536294] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7904.554168] Lustre: 251059:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7905.010082] LustreError: 247980:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7905.052444] LustreError: 247981:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7911.037386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7911.053086] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7911.154550] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7912.383148] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 7915.135132] LustreError: 247981:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7921.255400] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7922.720354] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7924.761820] Lustre: lustre-OST0001: Not available for connect from 192.168.201.13@tcp (stopping) [ 7926.758583] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7926.763447] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7926.786492] Lustre: Skipped 2 previous similar messages [ 7926.794573] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7929.822688] Lustre: lustre-OST0001: Not available for connect from 192.168.201.13@tcp (stopping) [ 7929.830342] Lustre: Skipped 1 previous similar message [ 7930.715742] Lustre: server umount lustre-OST0001 complete [ 7931.887745] LustreError: 249927:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7931.905886] LustreError: 249927:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7947.234989] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7947.237454] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7947.248141] Lustre: Skipped 1 previous similar message [ 7952.353896] LustreError: 249929:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7952.385063] LustreError: 249929:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 7953.626988] Lustre: server umount lustre-OST0000 complete [ 7957.481157] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7957.491923] LustreError: 238877:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000004ff2ad62 x1873616113395072/t0(0) o8->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7957.573671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7957.585866] Lustre: Skipped 4 previous similar messages [ 7964.017418] Lustre: server umount lustre-MDT0000 complete [ 7968.474152] LustreError: 247966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786819954 with bad export cookie 2347804086723809085 [ 7968.478514] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7968.483221] LustreError: 247966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 7974.995773] Lustre: server umount lustre-MDT0001 complete [ 7988.156829] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 7990.879794] Key type lgssc unregistered [ 7991.146574] LNet: 253097:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7991.156586] LNetError: 253097:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7991.176798] LNet: Removed LNI 192.168.201.113@tcp [ 7991.973904] Key type .llcrypt unregistered [ 7991.975929] Key type ._llcrypt unregistered [ 8031.511732] Key type ._llcrypt registered [ 8031.514332] Key type .llcrypt registered [ 8031.591891] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 8032.438758] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8032.631308] alg: No test for adler32 (adler32-zlib) [ 8033.700859] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [ 8033.925309] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 8035.599243] Key type lgssc registered [ 8036.724090] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8043.529569] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8043.547627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8044.963346] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8044.987739] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8045.218156] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8049.294891] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 8058.907776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8059.070170] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8059.109963] Lustre: 255303:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8059.525592] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8064.372521] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 8072.535350] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8079.212150] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8086.452129] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8086.758121] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8087.056880] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8093.381795] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 8095.896748] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8101.326035] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8157.666791] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8157.679709] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8157.701444] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8162.272771] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8162.290677] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8162.304207] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8163.734736] Lustre: server umount lustre-OST0000 complete [ 8169.061360] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8182.145837] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8182.259631] LustreError: 258891:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8182.270765] LustreError: 258891:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8182.410205] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8187.516639] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8187.538591] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8187.546633] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8187.567165] Lustre: Skipped 1 previous similar message [ 8188.191136] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 8196.969027] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8239.072595] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8239.084531] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8239.095937] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8239.121682] Lustre: Skipped 1 previous similar message [ 8239.137896] Lustre: Skipped 2 previous similar messages [ 8242.959605] Lustre: server umount lustre-OST0000 complete [ 8248.807140] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8248.821775] LustreError: Skipped 1 previous similar message [ 8248.827389] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8248.844453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8253.195639] Lustre: server umount lustre-MDT0000 complete [ 8254.432157] LustreError: 255317:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8254.456864] LustreError: 255317:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8256.861155] LustreError: 255298:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786820242 with bad export cookie 7069617935538112163 [ 8256.871575] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8257.257213] Lustre: server umount lustre-MDT0001 complete [ 8269.904873] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 8272.286963] Key type lgssc unregistered [ 8272.530787] LNet: 260732:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8272.537185] LNetError: 260732:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8273.574696] LNet: Removed LNI 192.168.201.113@tcp [ 8274.379943] Key type .llcrypt unregistered [ 8274.383124] Key type ._llcrypt unregistered [ 8300.808961] Key type ._llcrypt registered [ 8300.816358] Key type .llcrypt registered [ 8300.943530] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 8302.085662] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8302.158516] alg: No test for adler32 (adler32-zlib) [ 8303.311649] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [ 8303.739946] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 8305.407205] Key type lgssc registered [ 8306.513726] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8313.996653] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8314.005319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8315.408875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8319.939580] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 8325.604384] LustreError: 262150:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8325.634456] LustreError: 262150:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8328.275182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8328.536884] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8334.121404] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 8342.039516] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8348.992412] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8356.751500] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8357.140553] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8360.181743] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8364.104909] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 8371.922963] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8399.346714] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8399.350724] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8399.376280] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8399.381113] Lustre: Skipped 1 previous similar message [ 8403.990184] Lustre: server umount lustre-OST0000 complete [ 8409.569856] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8409.580978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8409.581423] Lustre: Skipped 2 previous similar messages [ 8409.606675] Lustre: Skipped 1 previous similar message [ 8414.029452] Lustre: server umount lustre-MDT0000 complete [ 8414.693871] LustreError: 262151:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8414.720644] LustreError: 262151:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8418.228473] LustreError: 262132:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786820404 with bad export cookie 13385576152845739739 [ 8418.253674] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8418.680572] Lustre: server umount lustre-MDT0001 complete [ 8431.558623] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 8434.667871] Key type lgssc unregistered [ 8434.903930] LNet: 266461:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8434.911462] LNetError: 266461:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8434.942241] LNet: Removed LNI 192.168.201.113@tcp [ 8435.744224] Key type .llcrypt unregistered [ 8435.745737] Key type ._llcrypt unregistered [ 8454.576850] Key type ._llcrypt registered [ 8454.579439] Key type .llcrypt registered [ 8454.717881] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 8456.005376] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8456.086520] alg: No test for adler32 (adler32-zlib) [ 8457.563404] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [ 8458.133063] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 8459.903124] Key type lgssc registered [ 8461.353763] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8469.668145] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8469.714202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8471.331546] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8476.556437] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 8481.772089] LustreError: 267685:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8481.792658] LustreError: 267685:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8486.524193] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8486.828349] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8491.699198] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 8499.931867] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8507.300395] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8516.409913] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8516.616475] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8517.676895] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8524.076319] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 8532.745293] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8546.301722] Lustre: server umount lustre-OST0000 complete [ 8551.395995] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8551.403768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8551.421067] Lustre: Skipped 1 previous similar message [ 8556.525517] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8556.536781] Lustre: Skipped 2 previous similar messages [ 8561.634379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8561.640337] Lustre: Skipped 1 previous similar message [ 8562.655272] Lustre: 266993:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786820532/real 1786820532] req@000000007f69e3bc x1873617033916928/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786820548 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8562.695208] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8563.743428] Lustre: 266996:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786820533/real 1786820533] req@00000000ffbcb751 x1873617033917184/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786820549 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8565.215191] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8565.845060] Lustre: server umount lustre-MDT0000 complete [ 8567.583171] Lustre: 266994:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786820537/real 1786820537] req@000000001ece5301 x1873617033918976/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786820553 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8567.795282] LustreError: 267685:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8567.815657] LustreError: 267685:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8570.315971] LustreError: 267667:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786820556 with bad export cookie 11697491684165036444 [ 8570.318354] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8570.339771] LustreError: 267667:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8571.393909] Lustre: server umount lustre-MDT0001 complete [ 8586.404868] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 8589.688575] Key type lgssc unregistered [ 8589.890979] LNet: 271381:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8589.897719] LNetError: 271381:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8590.949508] LNet: Removed LNI 192.168.201.113@tcp [ 8592.088938] Key type .llcrypt unregistered [ 8592.091250] Key type ._llcrypt unregistered [ 8621.490681] Key type ._llcrypt registered [ 8621.493359] Key type .llcrypt registered [ 8621.585609] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 8623.505579] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8623.526600] alg: No test for adler32 (adler32-zlib) [ 8624.843614] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [ 8625.293144] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 8627.064191] Key type lgssc registered [ 8628.704932] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8638.951074] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8639.001121] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8640.549809] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8645.198914] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 8650.724727] LustreError: 272813:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8650.753546] LustreError: 272813:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8654.321308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8654.846249] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8659.666363] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 8668.815989] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8673.755368] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8681.291331] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8681.611310] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8685.689933] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8689.037990] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 8697.064959] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8727.008114] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8727.023237] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8727.061787] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8731.106595] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8731.111911] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8731.122432] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8731.157919] Lustre: Skipped 1 previous similar message [ 8732.515476] Lustre: server umount lustre-OST0000 complete [ 8736.737160] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8736.744275] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8736.766853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8741.347493] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8741.353355] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8742.692190] Lustre: server umount lustre-MDT0000 complete [ 8746.477692] LustreError: 272813:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8746.495266] LustreError: 272813:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8746.621621] LustreError: 274655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786820732 with bad export cookie 7938469614215715928 [ 8746.629747] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8746.633729] LustreError: 274655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8747.039325] Lustre: server umount lustre-MDT0001 complete [ 8759.925603] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 8762.554305] Key type lgssc unregistered [ 8762.940537] LNet: 277122:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8762.948232] LNetError: 277122:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8762.967341] LNet: Removed LNI 192.168.201.113@tcp [ 8763.670485] Key type .llcrypt unregistered [ 8763.673232] Key type ._llcrypt unregistered [ 8783.267856] Key type ._llcrypt registered [ 8783.270654] Key type .llcrypt registered [ 8783.370983] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 8784.172441] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8784.264858] alg: No test for adler32 (adler32-zlib) [ 8785.205475] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [ 8785.425976] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 8787.047176] Key type lgssc registered [ 8788.022653] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8794.533138] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8794.544796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8796.076608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8800.456752] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 8806.376381] LustreError: 278361:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8806.389606] LustreError: 278361:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8808.764939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8809.042357] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8812.996269] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 8820.535259] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8826.624179] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8833.778295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8834.150067] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8839.691080] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 8841.337033] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8847.117433] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8864.227755] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8864.263860] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8864.278803] Lustre: Skipped 1 previous similar message [ 8869.299834] Lustre: server umount lustre-OST0000 complete [ 8874.468921] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8874.477387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8874.494015] Lustre: Skipped 2 previous similar messages [ 8874.510993] Lustre: Skipped 1 previous similar message [ 8879.394803] Lustre: server umount lustre-MDT0000 complete [ 8879.586173] LustreError: 278369:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8879.619068] LustreError: 278369:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8883.328926] LustreError: 278327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786820869 with bad export cookie 15917382666378271333 [ 8883.332800] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8883.345507] LustreError: 278327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8883.979886] Lustre: server umount lustre-MDT0001 complete [ 8898.740559] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 8901.772831] Key type lgssc unregistered [ 8902.060763] LNet: 282065:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8902.065431] LNetError: 282065:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8903.080654] LNet: Removed LNI 192.168.201.113@tcp [ 8904.045134] Key type .llcrypt unregistered [ 8904.051497] Key type ._llcrypt unregistered [ 8917.392675] Key type ._llcrypt registered [ 8917.396930] Key type .llcrypt registered [ 8917.515425] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8935.213655] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [ 8950.548802] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 8952.162952] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8952.212080] alg: No test for adler32 (adler32-zlib) [ 8953.342951] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [ 8953.594438] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 8955.359320] Key type lgssc registered [ 8956.553761] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8998.096413] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8998.155380] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8999.647829] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8999.750696] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8999.859671] Lustre: lustre-MDT0000: new disk, initializing [ 9000.064651] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9000.105453] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9004.810328] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 9016.105534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9016.228665] Lustre: 285621:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9016.299547] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9016.309390] Lustre: Skipped 1 previous similar message [ 9016.402109] Lustre: lustre-MDT0001: new disk, initializing [ 9016.509891] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9016.556738] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9016.578058] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9021.572130] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 9031.699515] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9038.165694] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9044.467248] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9044.683231] Lustre: lustre-OST0000: new disk, initializing [ 9044.685489] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9044.689664] Lustre: 287591:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9044.747981] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9051.264910] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 9052.282493] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9052.300191] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9052.464670] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9060.763964] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9065.479835] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9065.643265] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9069.733446] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9069.911483] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9073.122371] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9073.128509] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9073.151366] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9073.165307] Lustre: Skipped 1 previous similar message [ 9077.683072] Lustre: server umount lustre-OST0000 complete [ 9083.361968] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9083.365976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9083.383279] Lustre: Skipped 2 previous similar messages [ 9083.387094] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9083.397337] Lustre: Skipped 1 previous similar message [ 9088.481794] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9088.492976] Lustre: Skipped 1 previous similar message [ 9093.600765] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9093.612437] Lustre: Skipped 1 previous similar message [ 9096.159208] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9096.463060] Lustre: server umount lustre-MDT0000 complete [ 9099.563959] LustreError: 285614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786821085 with bad export cookie 11302447625943881944 [ 9099.565019] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9099.578875] LustreError: 285614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9099.901249] Lustre: server umount lustre-MDT0001 complete [ 9108.518542] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 15:11:33 (1786821093) [ 9115.189116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9115.873335] LustreError: 290003:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9115.896385] LustreError: 290003:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9115.989790] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9120.790496] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 9121.255251] LustreError: 290004:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9125.345655] LustreError: 290003:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9129.573282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9129.901891] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9134.307960] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 9142.157235] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9148.345774] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9155.052685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9155.328198] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9161.302261] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 9168.775914] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9185.249500] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9185.252591] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9185.266408] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9185.303127] Lustre: Skipped 1 previous similar message [ 9188.562742] Lustre: server umount lustre-OST0000 complete [ 9195.488476] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9195.492439] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9195.521195] Lustre: Skipped 1 previous similar message [ 9195.536664] Lustre: Skipped 2 previous similar messages [ 9198.641487] Lustre: server umount lustre-MDT0000 complete [ 9200.615174] LustreError: 290298:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9200.634361] LustreError: 290298:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9202.926601] LustreError: 289983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786821188 with bad export cookie 11302447625943882819 [ 9202.932114] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9202.934252] LustreError: 289983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9203.228620] Lustre: server umount lustre-MDT0001 complete [ 9217.959884] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 9220.952117] Key type lgssc unregistered [ 9221.283944] LNet: 293562:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9221.297643] LNetError: 293562:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9221.314788] LNet: Removed LNI 192.168.201.113@tcp [ 9222.337678] Key type .llcrypt unregistered [ 9222.340872] Key type ._llcrypt unregistered [ 9255.299295] Key type ._llcrypt registered [ 9255.305266] Key type .llcrypt registered [ 9255.540132] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [ 9269.907758] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 9271.225957] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9271.259891] alg: No test for adler32 (adler32-zlib) [ 9272.272217] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [ 9272.498632] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 9274.135174] Key type lgssc registered [ 9275.139818] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9324.302788] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9324.342673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9325.623752] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9325.668817] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9325.809423] Lustre: lustre-MDT0000: new disk, initializing [ 9325.924903] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9325.952422] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9330.967442] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 9341.289321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9341.432908] Lustre: 297394:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9341.482273] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9341.487157] Lustre: Skipped 1 previous similar message [ 9341.577375] Lustre: lustre-MDT0001: new disk, initializing [ 9341.655846] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9341.687900] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9341.701321] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9346.728445] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 9358.589814] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9367.201818] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9375.119024] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9375.410323] Lustre: lustre-OST2710: new disk, initializing [ 9375.414178] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9375.419499] Lustre: 299364:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9375.503173] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9382.465403] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9382.475803] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9382.595709] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9382.650413] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 9393.662772] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9400.974676] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9401.216599] Lustre: lustre-OST03e8: new disk, initializing [ 9401.219814] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9401.229282] Lustre: 300428:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9401.353639] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9408.173856] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 9411.750641] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9411.764888] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9411.918913] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9416.740251] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9424.054427] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9424.260684] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9428.633571] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9428.817850] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9433.405773] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9433.606795] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9438.329757] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9438.598148] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9442.792681] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9442.802772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9442.826818] Lustre: Skipped 1 previous similar message [ 9447.907172] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9447.917683] Lustre: Skipped 6 previous similar messages [ 9449.084036] Lustre: server umount lustre-MDT0000 complete [ 9452.626654] LustreError: 297386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786821438 with bad export cookie 10763350366433392446 [ 9452.634725] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9452.641218] LustreError: 297386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9452.909368] Lustre: server umount lustre-MDT0001 complete [ 9467.040615] Lustre: server umount lustre-OST2710 complete [ 9469.215895] Lustre: 295190:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786821439/real 1786821439] req@000000007f69e3bc x1873617887498112/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786821455 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9469.258800] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9469.279663] Lustre: Skipped 2 previous similar messages [ 9470.609421] Lustre: server umount lustre-OST03e8 complete [ 9490.901930] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [ 9502.166660] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 9538.918488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9539.129794] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9539.156178] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9539.285801] Lustre: lustre-MDT0000: new disk, initializing [ 9539.402320] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9539.432785] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9543.348703] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 9554.067648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9554.199714] Lustre: 305680:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9554.280222] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9554.285235] Lustre: Skipped 1 previous similar message [ 9554.360237] Lustre: lustre-MDT0001: new disk, initializing [ 9554.439363] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9554.474422] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9554.489079] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9559.011808] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 9569.099871] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9574.736437] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9580.839768] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9581.116536] Lustre: lustre-OST0000: new disk, initializing [ 9581.121423] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9581.135514] Lustre: 307648:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9581.248856] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9586.937715] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 9590.340333] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9590.351226] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9590.435298] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9596.275251] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9600.902820] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9601.094977] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9605.794298] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9606.036900] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9611.241049] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9611.250830] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9611.255316] LustreError: Skipped 1 previous similar message [ 9611.275857] Lustre: Skipped 1 previous similar message [ 9611.280414] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9614.645751] Lustre: server umount lustre-OST0000 complete [ 9620.963463] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9620.968987] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9620.988163] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9620.994920] Lustre: Skipped 1 previous similar message [ 9625.277122] Lustre: server umount lustre-MDT0000 complete [ 9628.966455] LustreError: 305671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786821614 with bad export cookie 10763350366433394119 [ 9628.977546] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9628.986716] LustreError: 305671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9629.412933] Lustre: server umount lustre-MDT0001 complete [ 9638.970573] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 15:20:24 (1786821624) [ 9640.536511] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9642.397956] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 15:20:27 (1786821627) [ 9645.575372] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 9672.189871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9672.529708] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9672.564942] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9672.819865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9677.723936] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 9687.115291] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9687.208216] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9687.237571] Lustre: 311502:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9691.641141] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 9713.007388] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9713.283386] Lustre: Denying initial registration attempt from nid 192.168.201.113@tcp, specified as failover [ 9713.295852] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9713.309607] LustreError: 312957:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 9713.325675] LustreError: 312957:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9713.337285] LustreError: 312957:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9713.413669] Lustre: server umount lustre-OST0000 complete [ 9713.417713] LustreError: 312957:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9720.801173] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9720.804811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9720.820261] Lustre: Skipped 2 previous similar messages [ 9720.827738] Lustre: Skipped 3 previous similar messages [ 9726.523514] Lustre: server umount lustre-MDT0000 complete [ 9730.303373] LustreError: 311494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786821716 with bad export cookie 10763350366433395008 [ 9730.305345] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9730.313017] LustreError: 311494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9730.644925] Lustre: server umount lustre-MDT0001 complete [ 9742.151354] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 9745.476542] Key type lgssc unregistered [ 9745.843822] LNet: 314079:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9745.850353] LNetError: 314079:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9745.872810] LNet: Removed LNI 192.168.201.113@tcp [ 9746.674158] Key type .llcrypt unregistered [ 9746.675759] Key type ._llcrypt unregistered [ 9760.024887] Key type ._llcrypt registered [ 9760.027359] Key type .llcrypt registered [ 9760.138576] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 9761.285399] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9761.489885] alg: No test for adler32 (adler32-zlib) [ 9762.563978] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [ 9762.758859] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 9764.415281] Key type lgssc registered [ 9765.470958] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9817.435407] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 9826.741357] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9826.769745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9828.049513] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9828.071634] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9828.288550] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9833.954873] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 9843.957366] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9844.121186] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9844.209533] Lustre: 316847:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9844.624994] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9850.307917] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 9858.261037] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9864.858828] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9885.101852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9885.312615] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9885.475180] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9892.849680] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 9901.565317] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9905.121819] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9905.139908] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9905.161767] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9905.632426] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9905.645686] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9905.677351] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9910.483490] Lustre: server umount lustre-OST0000 complete [ 9915.877842] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9915.907891] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9915.916563] Lustre: Skipped 1 previous similar message [ 9920.997332] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9921.005492] Lustre: Skipped 1 previous similar message [ 9925.094248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9929.183362] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9929.363848] Lustre: server umount lustre-MDT0000 complete [ 9932.963337] LustreError: 316838:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786821918 with bad export cookie 2566857724837102038 [ 9932.970580] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9932.980701] LustreError: 316838:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9933.420506] Lustre: server umount lustre-MDT0001 complete [ 9945.298862] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 9948.425715] Key type lgssc unregistered [ 9948.856855] LNet: 320422:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9948.873461] LNetError: 320422:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9948.888261] LNet: Removed LNI 192.168.201.113@tcp [ 9949.660759] Key type .llcrypt unregistered [ 9949.663275] Key type ._llcrypt unregistered [ 9975.708418] Key type ._llcrypt registered [ 9975.711601] Key type .llcrypt registered [ 9975.816855] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 9977.217783] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9977.340623] alg: No test for adler32 (adler32-zlib) [ 9978.366864] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [ 9978.643463] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 9980.255417] Key type lgssc registered [ 9981.466420] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9989.078353] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9989.103783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9990.647154] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9996.068234] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [10000.868882] LustreError: 321840:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10000.907770] LustreError: 321840:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10005.994379] LustreError: 321841:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10007.350195] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10007.854212] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10013.993987] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [10023.306144] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10031.498149] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10041.075783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10041.659355] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10048.992412] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [10056.408824] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10070.504316] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10070.519745] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10070.532109] Lustre: Skipped 1 previous similar message [10073.607707] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.13@tcp (stopping) [10073.613995] Lustre: Skipped 2 previous similar messages [10075.619188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10075.624827] Lustre: Skipped 2 previous similar messages [10078.693925] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.13@tcp (stopping) [10083.808125] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.13@tcp (stopping) [10083.811509] Lustre: Skipped 3 previous similar messages [10084.319123] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10084.511153] Lustre: server umount lustre-MDT0000 complete [10085.865127] LustreError: 321836:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10085.891625] LustreError: 321836:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10088.487736] LustreError: 321822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786822074 with bad export cookie 8454688356388443709 [10088.500352] LustreError: 321822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10088.506204] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10088.927089] LustreError: 321837:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10090.989150] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10090.999159] Lustre: Skipped 1 previous similar message [10093.984267] LustreError: 321835:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10093.987298] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10094.003397] LustreError: 321835:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10094.026384] Lustre: Skipped 2 previous similar messages [10094.984664] Lustre: server umount lustre-MDT0001 complete [10102.148293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10111.894594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10113.055176] LustreError: 325442:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [10113.065472] LustreError: 325442:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@00000000405cf31b x1873618627767808/t0(0) o250->MGC192.168.201.113@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1786822099 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10113.086348] LustreError: 325442:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [10113.509561] LustreError: 321149:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000483a78c5 x1873618627768192/t0(0) o250->MGC192.168.201.113@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10113.761228] LustreError: 323674:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10113.771458] LustreError: 323674:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10113.858104] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10113.938125] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10118.666506] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [10123.066880] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10128.332271] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10128.827806] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10134.003191] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10134.110605] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [10141.945803] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10143.559276] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10149.158358] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10150.986931] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10159.593390] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10159.600900] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10159.605809] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10159.605912] Lustre: Skipped 1 previous similar message [10159.628990] Lustre: Skipped 1 previous similar message [10161.657768] Lustre: server umount lustre-OST0000 complete [10169.826063] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10169.828295] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10169.850916] Lustre: Skipped 1 previous similar message [10180.063216] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10180.410256] Lustre: server umount lustre-MDT0000 complete [10184.165449] LustreError: 325459:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10184.178803] LustreError: 325459:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [10184.665375] LustreError: 324647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786822170 with bad export cookie 8454688356388451311 [10184.672157] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10184.683686] LustreError: 324647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10185.251862] Lustre: server umount lustre-MDT0001 complete [10198.631919] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [10201.424681] Key type lgssc unregistered [10201.663382] LNet: 328132:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10201.669625] LNetError: 328132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10202.732405] LNet: Removed LNI 192.168.201.113@tcp [10203.621801] Key type .llcrypt unregistered [10203.624625] Key type ._llcrypt unregistered [10227.253087] Key type ._llcrypt registered [10227.256918] Key type .llcrypt registered [10227.392787] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [10228.385335] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10228.412355] alg: No test for adler32 (adler32-zlib) [10229.441683] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [10229.602547] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [10231.279950] Key type lgssc registered [10232.280754] Lustre: Echo OBD driver; http://www.lustre.org/ [10239.652829] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10239.671518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10241.235947] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10246.421629] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [10251.747685] LustreError: 329550:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10251.784530] LustreError: 329550:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10255.658360] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10256.350418] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10263.393582] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [10272.504965] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10278.806893] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10287.072745] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10287.091760] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10287.105471] Lustre: Skipped 1 previous similar message [10292.192242] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10292.206320] Lustre: Skipped 1 previous similar message [10297.312671] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10297.325180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10297.342244] Lustre: Skipped 1 previous similar message [10297.656303] Lustre: server umount lustre-MDT0000 complete [10301.414089] LustreError: 329550:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10301.459610] LustreError: 329550:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10301.697688] LustreError: 329531:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786822287 with bad export cookie 14543012803709128133 [10301.703889] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10301.710826] LustreError: 329531:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10302.043524] Lustre: server umount lustre-MDT0001 complete [10318.070479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10318.337666] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10318.384591] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10318.644864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10324.183284] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [10334.562309] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10334.671930] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10334.692773] Lustre: 332141:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10334.913793] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10339.170228] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [10347.060994] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10353.745309] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10364.718351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10364.912875] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10364.922714] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10364.932470] LustreError: 334029:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [10364.941775] LustreError: 334029:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [10364.946164] LustreError: 334029:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10365.023954] Lustre: server umount lustre-OST0000 complete [10365.027764] LustreError: 334029:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10376.694995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10376.866232] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10377.016388] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10383.548272] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10383.572297] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [10391.869102] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10400.180127] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10400.311065] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10400.398771] Lustre: lustre-OST0001: new disk, initializing [10400.402101] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10400.408149] Lustre: 335571:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10400.488583] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10406.935931] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [10409.613229] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10409.641330] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10409.787951] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10415.868802] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10420.197810] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10420.203354] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10420.214175] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10420.229440] Lustre: Skipped 2 previous similar messages [10420.253364] Lustre: Skipped 1 previous similar message [10425.225424] Lustre: server umount lustre-OST0001 complete [10425.343275] LustreError: 336046:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10425.380127] LustreError: 336046:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10430.446498] LustreError: 334496:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10430.448927] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10430.486554] LustreError: 334496:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10430.526697] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10430.534306] Lustre: Skipped 1 previous similar message [10435.557501] LustreError: 334498:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10436.440431] Lustre: server umount lustre-OST0000 complete [10440.675870] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10440.691061] Lustre: Skipped 1 previous similar message [10440.694861] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10440.698438] Lustre: Skipped 3 previous similar messages [10447.053165] Lustre: server umount lustre-MDT0000 complete [10450.920275] LustreError: 334519:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10450.957865] LustreError: 334519:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10451.608976] LustreError: 332134:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786822437 with bad export cookie 14543012803709128686 [10451.612417] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10451.626802] LustreError: 332134:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10452.143350] Lustre: server umount lustre-MDT0001 complete [10465.128374] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [10468.104971] Key type lgssc unregistered [10468.474995] LNet: 337534:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10468.481631] LNetError: 337534:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10468.494920] LNet: Removed LNI 192.168.201.113@tcp [10469.436946] Key type .llcrypt unregistered [10469.448140] Key type ._llcrypt unregistered [10545.765804] Key type ._llcrypt registered [10545.768801] Key type .llcrypt registered [10545.881900] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [10563.672180] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [10565.498522] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10565.527871] alg: No test for adler32 (adler32-zlib) [10566.668330] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [10566.974856] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [10568.663175] Key type lgssc registered [10570.280920] Lustre: Echo OBD driver; http://www.lustre.org/ [10608.570604] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10608.605461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10610.020301] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10610.080382] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10610.257718] Lustre: lustre-MDT0000: new disk, initializing [10610.396469] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10610.423356] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10615.452809] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [10626.578428] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10626.816657] Lustre: 342471:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10626.930913] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10626.946873] Lustre: Skipped 1 previous similar message [10627.086750] Lustre: lustre-MDT0001: new disk, initializing [10627.271969] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10627.359249] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10627.402624] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10632.632584] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [10643.029953] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10649.090431] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10655.675627] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10655.954372] Lustre: lustre-OST0000: new disk, initializing [10655.957831] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10655.963724] Lustre: 344442:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10656.033565] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10662.137375] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [10662.551834] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10662.561777] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10662.719218] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10671.635367] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10676.411639] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10676.643181] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10681.264134] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10681.445940] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10688.480498] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10688.485977] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10688.491086] LustreError: Skipped 1 previous similar message [10688.492833] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10688.527341] Lustre: Skipped 1 previous similar message [10689.964260] Lustre: server umount lustre-OST0000 complete [10698.722783] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10698.731906] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10698.737415] Lustre: Skipped 1 previous similar message [10698.754759] Lustre: Skipped 2 previous similar messages [10700.643737] Lustre: server umount lustre-MDT0000 complete [10704.888532] LustreError: 343267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786822690 with bad export cookie 13183766922776044757 [10704.903946] LustreError: 343267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10704.904343] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10705.216541] Lustre: server umount lustre-MDT0001 complete [10715.930737] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 15:38:21 (1786822701) [10726.146983] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 15:38:30 (1786822710) [10736.114793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10736.570962] LustreError: 347197:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10736.593596] LustreError: 347197:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10736.648110] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10741.223207] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [10741.741121] LustreError: 347198:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10746.851017] LustreError: 347197:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10750.870486] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10751.208648] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10756.208569] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [10765.335336] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10772.275140] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10779.352624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10779.633794] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10785.456440] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [10793.955698] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10805.303373] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10807.114854] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10809.036874] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10810.932368] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10812.788550] Lustre: Failing over lustre-MDT0000 [10812.898186] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10812.906635] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10812.920763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10813.023601] Lustre: server umount lustre-MDT0000 complete [10816.518599] LustreError: 347193:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10817.517281] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10817.531461] Lustre: Skipped 1 previous similar message [10821.609576] LustreError: 349036:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10821.625038] LustreError: 349036:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10831.856333] LustreError: 347192:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10831.891740] LustreError: 347192:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [10832.594812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10832.728312] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10832.874571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [10832.995807] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10833.044596] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10837.268629] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [10838.001911] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10838.524151] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10838.582795] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [10838.650871] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10848.118690] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10850.334910] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10854.713335] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10856.596028] Lustre: Failing over lustre-MDT0000 [10856.935309] Lustre: server umount lustre-MDT0000 complete [10858.515899] LustreError: 347193:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10858.978408] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10858.980317] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10859.000816] Lustre: Skipped 2 previous similar messages [10874.831178] Lustre: 340695:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786822844/real 1786822844] req@000000007c1e3f5a x1873619244376960/t0(0) o400->MGC192.168.201.113@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786822860 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10874.863538] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10876.300809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10884.363529] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10884.440606] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10885.102564] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10888.779235] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [10889.714520] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10889.719965] Lustre: Skipped 2 previous similar messages [10889.767612] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [10889.823479] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10898.438791] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10899.911919] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10910.178320] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10910.180243] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10910.182047] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10910.182056] Lustre: Skipped 2 previous similar messages [10910.212276] Lustre: Skipped 1 previous similar message [10914.144494] Lustre: server umount lustre-MDT0000 complete [10915.298540] LustreError: 347490:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10915.323370] LustreError: 347490:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 25 previous similar messages [10919.202650] LustreError: 347177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786822905 with bad export cookie 13183766922776047466 [10919.223326] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10919.224158] LustreError: 347177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10919.694311] Lustre: server umount lustre-MDT0001 complete [10935.223775] Lustre: server umount lustre-OST0000 complete [10952.143758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10952.608763] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10957.186489] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [10966.388852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10971.693704] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [10980.633063] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10989.355406] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10998.872964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11006.234256] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [11014.416244] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11022.868955] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11032.035491] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11032.041987] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11032.060459] Lustre: Skipped 1 previous similar message [11032.064262] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11032.071104] Lustre: Skipped 2 previous similar messages [11037.456437] Lustre: server umount lustre-OST0000 complete [11047.449527] Lustre: server umount lustre-MDT0000 complete [11048.416124] LustreError: 353549:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11048.442585] LustreError: 353549:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [11051.167994] LustreError: 353530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786823037 with bad export cookie 13183766922776048208 [11051.176096] LustreError: 353530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11051.185310] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11051.528397] Lustre: server umount lustre-MDT0001 complete [11064.599221] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [11067.850568] Key type lgssc unregistered [11068.201888] LNet: 357105:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11068.216259] LNetError: 357105:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11069.294511] LNet: Removed LNI 192.168.201.113@tcp [11070.238123] Key type .llcrypt unregistered [11070.245624] Key type ._llcrypt unregistered [11100.157690] Key type ._llcrypt registered [11100.169480] Key type .llcrypt registered [11100.342609] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [11101.809660] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11101.972591] alg: No test for adler32 (adler32-zlib) [11103.105711] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [11103.505149] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [11105.263308] Key type lgssc registered [11106.452615] Lustre: Echo OBD driver; http://www.lustre.org/ [11114.526721] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11114.545595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11116.157055] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11120.805090] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [11126.754503] LustreError: 358625:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11126.769647] LustreError: 358625:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11128.996980] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11129.336609] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11134.902822] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [11144.360399] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11151.288495] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11158.816803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11159.132507] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11161.201994] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11166.123465] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [11173.453464] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11188.192966] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11188.207713] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11188.222700] Lustre: Skipped 2 previous similar messages [11188.236538] Lustre: Skipped 2 previous similar messages [11193.314905] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11193.322269] Lustre: Skipped 2 previous similar messages [11193.750807] Lustre: server umount lustre-MDT0000 complete [11198.152254] LustreError: 358605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786823184 with bad export cookie 3598768124397777884 [11198.161151] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11198.168526] LustreError: 358605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11198.433911] LustreError: 358620:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11198.434444] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11198.639645] Lustre: server umount lustre-MDT0001 complete [11204.834688] Lustre: server umount lustre-OST0000 complete [11218.317844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11218.764835] LustreError: 362355:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11218.779973] LustreError: 362355:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11218.842117] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11223.715742] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [11224.033172] LustreError: 362356:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11228.132414] LustreError: 362355:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11233.232503] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11233.606456] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11238.641441] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [11247.792633] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11253.959401] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11261.023639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11261.510240] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11263.611733] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11268.494408] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [11277.846730] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11294.698620] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11294.705528] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11294.714486] Lustre: Skipped 2 previous similar messages [11294.726073] Lustre: Skipped 2 previous similar messages [11296.789650] Lustre: server umount lustre-MDT0000 complete [11299.835345] LustreError: 362356:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11299.856730] LustreError: 362356:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11301.574915] LustreError: 362336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786823287 with bad export cookie 3598768124397778990 [11301.610038] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11301.994322] Lustre: server umount lustre-MDT0001 complete [11317.063618] Lustre: server umount lustre-OST0000 complete [11333.976074] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11334.247100] LustreError: 366268:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11334.334572] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11340.144861] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [11350.355869] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11356.871631] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [11365.554239] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11372.788299] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11380.163147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11380.601746] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11380.611683] Lustre: Skipped 1 previous similar message [11387.199242] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [11395.895061] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11404.227779] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11414.512218] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11414.529812] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11414.535677] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11418.764995] Lustre: server umount lustre-OST0000 complete [11422.688848] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11422.695614] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11422.709667] Lustre: Skipped 1 previous similar message [11422.715919] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11422.723585] Lustre: Skipped 1 previous similar message [11429.073777] Lustre: server umount lustre-MDT0000 complete [11429.857278] LustreError: 367209:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11429.888704] LustreError: 367209:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [11433.483225] LustreError: 366250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786823419 with bad export cookie 3598768124397780012 [11433.497909] LustreError: 366250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11433.502387] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11433.811559] Lustre: server umount lustre-MDT0001 complete [11445.014537] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [11447.731737] Key type lgssc unregistered [11448.075893] LNet: 369833:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11448.083088] LNetError: 369833:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11448.103849] LNet: Removed LNI 192.168.201.113@tcp [11448.995109] Key type .llcrypt unregistered [11449.000551] Key type ._llcrypt unregistered [11475.608526] Key type ._llcrypt registered [11475.611308] Key type .llcrypt registered [11475.696339] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [11476.710456] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11476.748081] alg: No test for adler32 (adler32-zlib) [11477.838381] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [11478.172754] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [11479.823164] Key type lgssc registered [11480.859915] Lustre: Echo OBD driver; http://www.lustre.org/ [11487.696266] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11487.704413] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11487.713417] LustreError: 371291:0:(osd_handler.c:8553:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11487.742651] LustreError: 371291:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11487.747554] LustreError: 371291:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11487.755431] LustreError: 371291:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11487.761955] LustreError: 371291:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11494.740508] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11494.751296] LustreError: 371688:0:(osd_handler.c:8553:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11494.772082] LustreError: 371688:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11494.778167] LustreError: 371688:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11494.783553] LustreError: 371688:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11494.791055] LustreError: 371688:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11515.152729] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [11518.215029] Key type lgssc unregistered [11518.561765] LNet: 372694:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11518.567143] LNetError: 372694:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11518.600230] LNet: Removed LNI 192.168.201.113@tcp [11519.667569] Key type .llcrypt unregistered [11519.677776] Key type ._llcrypt unregistered [11539.849995] Key type ._llcrypt registered [11539.856366] Key type .llcrypt registered [11540.020643] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [11557.382500] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [11558.663322] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11558.815692] alg: No test for adler32 (adler32-zlib) [11559.788203] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [11559.989463] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [11561.671248] Key type lgssc registered [11562.774690] Lustre: Echo OBD driver; http://www.lustre.org/ [11607.318978] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11607.356398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11608.744573] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11608.780573] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11608.929805] Lustre: lustre-MDT0000: new disk, initializing [11609.065446] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11609.082027] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11613.255218] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [11625.307371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11625.440234] Lustre: 375853:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11625.506584] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11625.510993] Lustre: Skipped 1 previous similar message [11625.588567] Lustre: lustre-MDT0001: new disk, initializing [11625.693320] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11625.740870] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11625.762676] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11630.966608] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [11641.752043] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11648.070431] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11655.315156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11655.678466] Lustre: lustre-OST0000: new disk, initializing [11655.681476] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11655.686843] Lustre: 377822:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11655.797430] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11661.364105] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [11663.927120] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11663.939334] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11664.084919] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11671.465152] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11675.829853] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11676.092151] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11681.253521] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11681.459658] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11684.838692] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11684.841077] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11684.861492] Lustre: Skipped 1 previous similar message [11684.863290] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11689.926205] Lustre: server umount lustre-OST0000 complete [11695.076273] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11695.079216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11695.110145] Lustre: Skipped 1 previous similar message [11695.128533] Lustre: Skipped 2 previous similar messages [11700.192266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11700.211425] Lustre: Skipped 1 previous similar message [11704.293364] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11704.309180] Lustre: Skipped 1 previous similar message [11708.383173] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11708.543207] Lustre: server umount lustre-MDT0000 complete [11712.188689] LustreError: 375845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786823698 with bad export cookie 15871389387103256803 [11712.194627] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11712.201912] LustreError: 375845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11712.510585] Lustre: server umount lustre-MDT0001 complete [11721.853597] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11723.588380] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 15:55:08 (1786823708) [11731.153846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11731.849238] LustreError: 380285:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11731.859340] LustreError: 380285:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11731.957740] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11736.906783] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [11737.069197] LustreError: 380286:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11741.155709] LustreError: 380285:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11745.602127] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11745.828811] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11750.783590] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [11758.718518] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11765.644270] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11772.565181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11773.000091] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11780.297398] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [11789.090443] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11797.493463] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11797.860516] Lustre: lustre-OST0001: new disk, initializing [11797.870248] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11797.892509] Lustre: 383123:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11798.086587] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11804.842874] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [11806.916549] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11806.926904] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11807.016415] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11814.878840] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11822.072573] Lustre: lustre-OST0001: Not available for connect from 192.168.201.13@tcp (stopping) [11822.560619] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11822.562584] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11822.571959] LustreError: Skipped 1 previous similar message [11822.588619] Lustre: Skipped 1 previous similar message [11825.261944] Lustre: server umount lustre-OST0001 complete [11827.174779] LustreError: 382105:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11832.815537] LustreError: 382105:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11832.836819] LustreError: 382105:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11843.045384] LustreError: 382106:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11843.047792] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11843.072699] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11843.072708] Lustre: Skipped 2 previous similar messages [11843.073733] LustreError: 382106:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11843.106717] Lustre: Skipped 1 previous similar message [11856.351127] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11856.481597] Lustre: server umount lustre-OST0000 complete [11863.522446] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11863.534355] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11863.547894] Lustre: Skipped 6 previous similar messages [11863.550438] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11866.402157] Lustre: server umount lustre-MDT0000 complete [11868.649292] LustreError: 382726:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11868.673838] LustreError: 382726:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [11871.048623] LustreError: 380266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786823857 with bad export cookie 15871389387103257685 [11871.058536] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11877.460994] Lustre: server umount lustre-MDT0001 complete [11892.110403] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [11897.002484] Key type lgssc unregistered [11897.432633] LNet: 385114:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11897.447945] LNetError: 385114:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11897.474312] LNet: Removed LNI 192.168.201.113@tcp [11898.286222] Key type .llcrypt unregistered [11898.290232] Key type ._llcrypt unregistered [11952.942611] Key type ._llcrypt registered [11952.945431] Key type .llcrypt registered [11953.084347] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [11954.522238] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11954.599641] alg: No test for adler32 (adler32-zlib) [11955.598634] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [11955.786376] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [11957.415345] Key type lgssc registered [11958.643608] Lustre: Echo OBD driver; http://www.lustre.org/ [11967.205289] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11967.222814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11968.478875] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11968.515894] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11968.676833] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11973.275423] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [11982.203127] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11982.328606] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11982.370045] Lustre: 387929:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11982.670071] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11987.565727] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [11995.772518] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12002.139647] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12005.348741] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12005.366194] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12005.383109] Lustre: Skipped 1 previous similar message [12010.474550] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12010.481353] Lustre: Skipped 1 previous similar message [12010.918792] Lustre: server umount lustre-MDT0000 complete [12014.510900] LustreError: 387923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786824000 with bad export cookie 6230820699809167236 [12014.526824] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12014.534944] LustreError: 387923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12014.873814] Lustre: server umount lustre-MDT0001 complete [12025.708750] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 16:00:10 (1786824010) [12032.280205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12032.643373] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12036.949651] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [12038.114148] LustreError: 390445:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12038.146957] LustreError: 390445:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12043.231947] LustreError: 390444:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12045.776709] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12046.104857] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12049.898642] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [12057.357672] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12063.990833] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12070.998200] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12071.230457] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [12071.512084] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12078.121964] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [12086.610028] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12094.062568] Lustre: Permanently deactivating lustre-OST0000 [12094.083253] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [12104.242265] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12105.640114] LustreError: 393489:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [12105.648829] LustreError: 393489:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12105.654379] LustreError: 393489:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12108.203448] LustreError: 393537:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12108.210256] LustreError: 393537:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12116.096891] Lustre: server umount lustre-OST0000 complete [12118.448935] LustreError: 393788:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12118.459550] LustreError: 393788:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12123.105034] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12123.114054] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12123.131167] Lustre: Skipped 1 previous similar message [12123.141189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12125.156570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12125.162527] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12130.277203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12130.303906] Lustre: Skipped 2 previous similar messages [12131.417518] Lustre: server umount lustre-MDT0000 complete [12135.400750] LustreError: 390445:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12135.434816] LustreError: 390445:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12135.487215] LustreError: 393145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786824121 with bad export cookie 6230820699809167817 [12135.498969] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12135.500449] LustreError: 393145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [12145.386184] Lustre: server umount lustre-MDT0001 complete [12155.886592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12161.021539] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [12164.673216] LustreError: 394947:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12172.601833] Lustre: 395288:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.113@tcp [12175.194949] Lustre: server umount MGS complete [12188.297412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12189.054243] LustreError: 396012:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12189.073830] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12189.077857] Lustre: Skipped 1 previous similar message [12189.124109] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12193.592650] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [12194.278647] LustreError: 396013:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12201.810415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12202.055223] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12206.850611] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [12215.852307] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12222.313939] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12224.268357] Lustre: Permanently reactivating lustre-OST0000 [12224.282813] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12224.286014] Lustre: Skipped 2 previous similar messages [12252.130377] LustreError: 397791:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12258.762466] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12258.986913] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12258.994812] Lustre: Skipped 1 previous similar message [12264.343454] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [12272.057542] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12277.737959] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12277.741459] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12277.744991] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12277.757020] Lustre: Skipped 1 previous similar message [12277.792487] LustreError: Skipped 1 previous similar message [12283.778360] Lustre: server umount lustre-OST0000 complete [12287.970798] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12293.633589] Lustre: server umount lustre-MDT0000 complete [12297.272155] LustreError: 395993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786824283 with bad export cookie 6230820699809168867 [12297.278472] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12297.284649] LustreError: 395993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12297.627050] Lustre: server umount lustre-MDT0001 complete [12309.954921] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [12312.919343] Key type lgssc unregistered [12313.220646] LNet: 400378:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12313.226962] LNetError: 400378:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12313.242396] LNet: Removed LNI 192.168.201.113@tcp [12314.348816] Key type .llcrypt unregistered [12314.353742] Key type ._llcrypt unregistered [12336.771247] Key type ._llcrypt registered [12336.773666] Key type .llcrypt registered [12336.997083] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [12356.086417] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [12357.206574] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12357.375043] alg: No test for adler32 (adler32-zlib) [12358.531892] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [12358.745388] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [12360.423145] Key type lgssc registered [12362.132705] Lustre: Echo OBD driver; http://www.lustre.org/ [12401.467618] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12401.514826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12403.055611] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12403.104912] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12403.249866] Lustre: lustre-MDT0000: new disk, initializing [12403.421305] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12403.483556] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12408.925317] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [12422.661885] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12422.799959] Lustre: 403552:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12422.855547] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12422.858485] Lustre: Skipped 1 previous similar message [12422.912603] Lustre: lustre-MDT0001: new disk, initializing [12423.043240] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12423.085823] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12423.113533] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12429.041546] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [12441.150326] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12447.912055] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12455.379767] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12455.873894] Lustre: lustre-OST0000: new disk, initializing [12455.877664] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12455.885785] Lustre: 405523:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12456.010387] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12463.489166] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [12465.777320] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12465.785786] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12465.888123] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12473.559588] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12478.830869] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12479.205381] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12484.853791] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12485.042885] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12491.748961] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12491.749530] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12491.766735] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12491.780586] Lustre: Skipped 1 previous similar message [12491.792928] Lustre: Skipped 1 previous similar message [12493.375806] Lustre: server umount lustre-OST0000 complete [12499.937871] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12499.945662] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12499.962668] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12501.985472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12501.991985] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12507.107308] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12507.114931] Lustre: Skipped 2 previous similar messages [12511.201549] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12511.201697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12511.224346] Lustre: Skipped 1 previous similar message [12511.466413] Lustre: server umount lustre-MDT0000 complete [12514.709327] LustreError: 403544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786824500 with bad export cookie 9303319883103816371 [12514.713459] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12514.716204] LustreError: 403544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12515.077192] Lustre: server umount lustre-MDT0001 complete [12525.168775] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12527.440588] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 16:08:32 (1786824512) [12534.476312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12534.972566] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12539.781854] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [12540.388101] LustreError: 407986:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12540.432980] LustreError: 407986:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12544.490300] LustreError: 407985:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12548.769962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12549.041590] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12552.772939] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [12562.203155] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12569.943809] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12579.016811] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12579.287685] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12586.951690] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [12595.269609] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12597.169328] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12599.588887] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12605.920797] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12605.925359] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12605.933862] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12609.001162] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12610.890503] Lustre: server umount lustre-OST0000 complete [12615.652025] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12615.670860] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12615.693438] Lustre: Skipped 1 previous similar message [12620.968676] Lustre: server umount lustre-MDT0000 complete [12624.353540] LustreError: 407986:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12624.378511] LustreError: 407986:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12625.486605] LustreError: 410961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786824611 with bad export cookie 9303319883103817253 [12625.495013] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12625.498825] LustreError: 410961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12625.877440] Lustre: server umount lustre-MDT0001 complete [12637.529207] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [12640.087093] Key type lgssc unregistered [12640.412782] LNet: 411715:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12640.421647] LNetError: 411715:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12640.433857] LNet: Removed LNI 192.168.201.113@tcp [12641.092600] Key type .llcrypt unregistered [12641.094581] Key type ._llcrypt unregistered [12666.570817] Key type ._llcrypt registered [12666.573390] Key type .llcrypt registered [12666.682508] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [12666.983321] Key type .llcrypt unregistered [12666.986454] Key type ._llcrypt unregistered [12685.675196] Key type ._llcrypt registered [12685.683929] Key type .llcrypt registered [12685.780323] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [12686.857770] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12687.026209] alg: No test for adler32 (adler32-zlib) [12688.166470] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [12688.450977] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [12690.189413] Key type lgssc registered [12691.286135] Lustre: Echo OBD driver; http://www.lustre.org/ [12699.463706] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12699.489327] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12700.825898] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12705.314834] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [12710.375592] LustreError: 413932:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12710.400223] LustreError: 413932:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12712.971480] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12713.200559] LustreError: 413933:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12713.221949] LustreError: 413933:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12713.228860] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12718.562377] LustreError: 413929:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12718.588680] LustreError: 413929:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12719.292618] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [12723.692935] LustreError: 413932:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12723.719101] LustreError: 413932:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12725.746779] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12727.195996] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12730.854359] LustreError: 413933:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12730.870259] LustreError: 413933:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12733.731260] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12734.126069] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12738.589092] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [12748.662416] Lustre: server umount lustre-OST0000 complete [12753.904931] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12753.917561] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12753.944744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12754.912802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12754.921174] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12760.035104] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12760.051809] Lustre: Skipped 2 previous similar messages [12762.257159] Lustre: server umount lustre-MDT0000 complete [12765.153970] LustreError: 413932:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12765.190763] LustreError: 413932:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12765.471332] Lustre: 413246:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786824735/real 1786824735] req@000000000afe0216 x1873621469390336/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786824751 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12765.492684] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12767.324329] LustreError: 413914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786824753 with bad export cookie 11893769775646211633 [12767.338710] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12767.340657] LustreError: 413914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12767.976360] Lustre: server umount lustre-MDT0001 complete [12780.556378] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [12783.380262] Key type lgssc unregistered [12783.687646] LNet: 417083:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12783.694561] LNetError: 417083:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12783.726648] LNet: Removed LNI 192.168.201.113@tcp [12784.711841] Key type .llcrypt unregistered [12784.714760] Key type ._llcrypt unregistered [12809.726333] Key type ._llcrypt registered [12809.728806] Key type .llcrypt registered [12809.840451] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [12811.341261] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12811.403716] alg: No test for adler32 (adler32-zlib) [12812.510658] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [12812.773227] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [12814.407171] Key type lgssc registered [12815.344205] Lustre: Echo OBD driver; http://www.lustre.org/ [12823.809504] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12823.854498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12839.519650] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12844.639364] LustreError: 418474:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.113@tcp: failed processing log, type 4: rc = -110 [12871.263759] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12876.840409] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [12884.244688] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12891.736929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12892.218778] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12892.291107] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12895.843899] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [12897.252273] LustreError: 418499:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12897.271599] LustreError: 418499:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12902.377346] LustreError: 418500:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12904.759525] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12905.355365] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12909.931785] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [12917.472992] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12923.433609] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12931.047551] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12931.056089] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12931.070615] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12932.078921] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12932.104387] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12932.931095] Lustre: server umount lustre-OST0000 complete [12937.188530] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12937.202433] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12937.208977] Lustre: Skipped 2 previous similar messages [12939.014953] Lustre: server umount lustre-MDT0000 complete [12942.308628] LustreError: 420575:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12942.334776] LustreError: 420575:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12943.139207] LustreError: 418481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786824929 with bad export cookie 11712838445413975639 [12943.146505] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12943.151386] LustreError: 418481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12943.457766] Lustre: server umount lustre-MDT0001 complete [12955.664569] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [12958.108880] Key type lgssc unregistered [12958.410338] LNet: 422043:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12958.424493] LNetError: 422043:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12958.446796] LNet: Removed LNI 192.168.201.113@tcp [12959.358199] Key type .llcrypt unregistered [12959.360760] Key type ._llcrypt unregistered [12982.785634] Key type ._llcrypt registered [12982.788582] Key type .llcrypt registered [12982.883677] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [12983.839029] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12983.880379] alg: No test for adler32 (adler32-zlib) [12984.948854] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [12985.153366] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [12986.784991] Key type lgssc registered [12987.794870] Lustre: Echo OBD driver; http://www.lustre.org/ [12994.019940] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12994.040656] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12995.436688] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12999.422016] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [13005.792876] LustreError: 423459:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13005.824133] LustreError: 423459:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13007.174910] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13007.481775] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13011.274403] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [13020.138313] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13026.791208] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13034.562791] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13035.046670] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13040.193651] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [13041.070667] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [13050.056786] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13057.029401] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.13@tcp (stopping) [13057.514662] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13057.532706] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13057.534674] Lustre: Skipped 2 previous similar messages [13057.555638] Lustre: Skipped 2 previous similar messages [13060.853674] Lustre: server umount lustre-MDT0000 complete [13062.114501] LustreError: 423454:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13063.722728] LustreError: 423439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.13@tcp arrived at 1786825049 with bad export cookie 16148328570842878695 [13063.738031] LustreError: 423439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13065.550699] LustreError: 423440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786825051 with bad export cookie 16148328570842877806 [13065.553417] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13065.558312] LustreError: 423440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13067.744690] LustreError: 424891:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13067.746557] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13067.782356] LustreError: 424891:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [13067.796130] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13070.756223] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13070.758831] LustreError: 424186:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13070.786311] LustreError: 424186:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13071.716079] Lustre: server umount lustre-OST0000 complete [13075.874137] LustreError: 423455:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13075.903566] LustreError: 423455:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13078.335856] Lustre: server umount lustre-MDT0001 complete [13091.711833] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [13095.626758] Key type lgssc unregistered [13095.948790] LNet: 427149:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13095.967093] LNetError: 427149:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13095.990883] LNet: Removed LNI 192.168.201.113@tcp [13096.871636] Key type .llcrypt unregistered [13096.875065] Key type ._llcrypt unregistered [13129.549973] Key type ._llcrypt registered [13129.555448] Key type .llcrypt registered [13129.682642] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [13130.950537] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13131.008893] alg: No test for adler32 (adler32-zlib) [13132.066654] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [13132.372425] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [13134.079821] Key type lgssc registered [13135.423881] Lustre: Echo OBD driver; http://www.lustre.org/ [13142.799448] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13142.818449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13144.266438] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13149.055363] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [13154.800499] LustreError: 428567:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13154.835265] LustreError: 428567:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13158.172941] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13158.541016] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13164.684195] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [13175.036082] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13181.686571] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13188.912423] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13189.239506] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13190.253056] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13195.228232] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [13203.138562] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13209.062683] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13209.082441] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13209.088691] Lustre: Skipped 1 previous similar message [13210.118072] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.13@tcp (stopping) [13213.257747] Lustre: server umount lustre-MDT0001 complete [13214.189109] LustreError: 428564:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13214.219172] LustreError: 428564:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13215.209799] LustreError: 429664:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13219.302502] LustreError: 428564:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13219.329694] LustreError: 428564:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13221.350590] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13221.362584] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13221.383189] Lustre: Skipped 1 previous similar message [13221.388468] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13224.417812] LustreError: 428567:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13224.420831] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13226.073614] Lustre: server umount lustre-OST0000 complete [13230.412114] Lustre: server umount lustre-MDT0000 complete [13248.475667] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [13251.926168] Key type lgssc unregistered [13252.240079] LNet: 432255:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13252.244603] LNetError: 432255:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13252.279445] LNet: Removed LNI 192.168.201.113@tcp [13253.352538] Key type .llcrypt unregistered [13253.358198] Key type ._llcrypt unregistered [13278.962579] Key type ._llcrypt registered [13278.964651] Key type .llcrypt registered [13279.048538] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [13294.622599] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [13295.760756] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13295.917137] alg: No test for adler32 (adler32-zlib) [13296.957619] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [13297.183562] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [13298.911174] Key type lgssc registered [13300.009899] Lustre: Echo OBD driver; http://www.lustre.org/ [13343.136798] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [13358.349206] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13358.385151] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13359.647478] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13359.681475] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13359.775132] Lustre: lustre-MDT0000: new disk, initializing [13359.872464] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13359.885261] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13364.523645] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [13379.274855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13379.553481] Lustre: lustre-OST0000: new disk, initializing [13379.556553] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13379.559440] Lustre: Skipped 1 previous similar message [13379.571205] Lustre: 437291:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13379.647974] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13386.375892] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13386.383372] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13386.418933] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13386.683652] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [13395.929371] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13410.773283] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13410.888452] Lustre: 436274:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13410.946362] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13411.003707] Lustre: lustre-MDT0001: new disk, initializing [13411.089670] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13411.127313] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13411.136198] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13414.937960] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [13423.670328] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13423.888368] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13428.577561] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13428.832188] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13433.844427] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.13@tcp (stopping) [13434.850050] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13434.863781] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13434.881255] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13438.945583] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.13@tcp (stopping) [13438.957102] Lustre: Skipped 2 previous similar messages [13439.538895] Lustre: server umount lustre-MDT0001 complete [13445.600786] LustreError: 436281:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13445.629686] LustreError: 436281:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [13450.729768] LustreError: 437529:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13450.750613] LustreError: 437529:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13455.844436] LustreError: 437529:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13455.857159] LustreError: 437529:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13458.400245] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13458.409052] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13458.422098] Lustre: Skipped 1 previous similar message [13458.428319] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13460.972071] LustreError: 436285:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13462.735806] Lustre: server umount lustre-OST0000 complete [13466.088225] LustreError: 436284:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13473.068610] Lustre: server umount lustre-MDT0000 complete [13486.255548] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [13489.150827] Key type lgssc unregistered [13489.587681] LNet: 441040:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13489.599830] LNetError: 441040:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13489.633589] LNet: Removed LNI 192.168.201.113@tcp [13490.779094] Key type .llcrypt unregistered [13490.780296] Key type ._llcrypt unregistered [13532.482519] Key type ._llcrypt registered [13532.488378] Key type .llcrypt registered [13532.606450] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [13532.843464] Key type .llcrypt unregistered [13532.848190] Key type ._llcrypt unregistered [13548.302724] Key type ._llcrypt registered [13548.306805] Key type .llcrypt registered [13548.456300] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [13550.764313] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13550.802314] alg: No test for adler32 (adler32-zlib) [13551.856750] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [13552.128114] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [13553.807184] Key type lgssc registered [13555.019619] Lustre: Echo OBD driver; http://www.lustre.org/ [13585.230415] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13585.256610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13586.663977] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13586.718055] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13586.800566] Lustre: lustre-MDT0000: new disk, initializing [13586.897057] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13586.930240] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13591.699431] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [13602.594153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13602.764383] Lustre: 444423:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13602.878448] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13602.887987] Lustre: Skipped 1 previous similar message [13602.969805] Lustre: lustre-MDT0001: new disk, initializing [13603.109261] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13603.161148] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13603.178059] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13607.377092] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [13616.380636] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13622.316432] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13629.983441] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13630.247481] Lustre: lustre-OST0000: new disk, initializing [13630.257966] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13630.278802] Lustre: 446396:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13630.429678] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13636.280501] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [13637.684031] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13637.689839] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13637.848288] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13647.168238] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13658.599704] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13658.610565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13658.621412] Lustre: Skipped 2 previous similar messages [13658.649093] Lustre: Skipped 2 previous similar messages [13662.539419] Lustre: server umount lustre-MDT0000 complete [13666.661692] LustreError: 446394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786825652 with bad export cookie 17227316903803354305 [13666.669293] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13666.674856] LustreError: 446394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13667.416132] Lustre: server umount lustre-MDT0001 complete [13682.179602] Lustre: server umount lustre-OST0000 complete [13693.645300] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 16:27:58 (1786825678) [13701.544130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13701.999337] LustreError: 448425:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13702.030923] LustreError: 448425:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [13702.120030] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13706.990305] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [13707.236346] LustreError: 448426:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13707.273962] LustreError: 448426:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13712.359017] LustreError: 448425:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13716.847604] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13717.446600] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13723.806873] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [13733.897819] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13740.347427] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13760.557236] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13760.823419] Lustre: Found index 0 for lustre-OST0000, updating log [13761.036259] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13768.261041] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [13773.302196] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13777.705771] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13783.521200] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13783.528548] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13783.543085] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13783.567725] Lustre: Skipped 1 previous similar message [13784.620572] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13784.631978] Lustre: Skipped 1 previous similar message [13789.358778] Lustre: server umount lustre-OST0000 complete [13809.632598] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13809.648091] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13809.676983] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13809.684770] Lustre: Skipped 1 previous similar message [13812.996253] Lustre: server umount lustre-MDT0000 complete [13816.800746] LustreError: 448422:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13816.815787] LustreError: 448422:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13817.421567] LustreError: 448405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786825803 with bad export cookie 17227316903803355705 [13817.428608] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13817.870194] Lustre: server umount lustre-MDT0001 complete [13828.061140] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 16:30:13 (1786825813) [13829.663822] LNet: 1 local NIs in recovery (showing 1): 192.168.201.113@tcp [14450.154176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14450.650964] LustreError: 454838:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14450.827986] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14455.102420] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [14456.297168] LustreError: 454839:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14460.392399] LustreError: 454838:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14465.181692] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14465.753149] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14470.972554] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [14480.029944] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14488.446805] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14506.530240] LNet: 1 local NIs in recovery (showing 1): 192.168.201.113@tcp [14510.759822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14511.146751] Lustre: Found index 0 for lustre-OST0000, updating log [14511.595213] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14519.221160] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [14527.138394] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14530.731913] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14530.754773] Lustre: Skipped 1 previous similar message [14530.765874] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14530.768291] Lustre: Skipped 4 previous similar messages [14535.955576] Lustre: server umount lustre-OST0000 complete [14541.156544] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14541.164597] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14541.169256] Lustre: Skipped 1 previous similar message [14542.176653] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14542.189111] Lustre: Skipped 1 previous similar message [14544.229658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14544.238138] Lustre: Skipped 3 previous similar messages [14546.664227] Lustre: server umount lustre-MDT0000 complete [14547.300138] LustreError: 454839:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14547.337492] LustreError: 454839:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14550.326450] LustreError: 457642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786826536 with bad export cookie 17227316903803356790 [14550.327458] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14550.335350] LustreError: 457642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [14550.691802] Lustre: server umount lustre-MDT0001 complete [14566.943920] LNet: 1 local NIs in recovery (showing 1): 192.168.201.113@tcp [14583.097730] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [14594.352486] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [14632.255525] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 16:43:37 (1786826617) [14650.713719] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [14653.123433] Key type lgssc unregistered [14653.452277] LNet: 461456:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14653.462651] LNetError: 461456:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14771.236878] LNet: Removed LNI 192.168.201.113@tcp [14772.330073] Key type .llcrypt unregistered [14772.333447] Key type ._llcrypt unregistered [14791.441631] Key type ._llcrypt registered [14791.443910] Key type .llcrypt registered [14791.560090] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [14793.294393] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14793.340267] alg: No test for adler32 (adler32-zlib) [14794.474769] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [14794.841798] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [14796.553581] Key type lgssc registered [14797.978529] Lustre: Echo OBD driver; http://www.lustre.org/ [14802.937839] LNet: Added LNI 192.168.201.113@tcp100 [8/256/0/180] [14803.890561] LNet: Added LNI 192.168.201.113@tcp101 [8/256/0/180] [14804.947352] LNet: Added LNI 192.168.201.113@tcp102 [8/256/0/180] [14805.912059] LNet: Added LNI 192.168.201.113@tcp103 [8/256/0/180] [14806.986020] LNet: Added LNI 192.168.201.113@tcp104 [8/256/0/180] [14807.977314] LNet: Added LNI 192.168.201.113@tcp105 [8/256/0/180] [14808.920369] LNet: Added LNI 192.168.201.113@tcp106 [8/256/0/180] [14810.070314] LNet: Added LNI 192.168.201.113@tcp107 [8/256/0/180] [14811.317900] LNet: Added LNI 192.168.201.113@tcp108 [8/256/0/180] [14812.410963] LNet: Added LNI 192.168.201.113@tcp109 [8/256/0/180] [14813.327751] LNet: Added LNI 192.168.201.113@tcp110 [8/256/0/180] [14814.236964] LNet: Added LNI 192.168.201.113@tcp111 [8/256/0/180] [14815.181942] LNet: Added LNI 192.168.201.113@tcp112 [8/256/0/180] [14816.107919] LNet: Added LNI 192.168.201.113@tcp113 [8/256/0/180] [14817.044816] LNet: Added LNI 192.168.201.113@tcp114 [8/256/0/180] [14818.233182] LNet: Added LNI 192.168.201.113@tcp115 [8/256/0/180] [14819.313949] LNet: Added LNI 192.168.201.113@tcp116 [8/256/0/180] [14820.460942] LNet: Added LNI 192.168.201.113@tcp117 [8/256/0/180] [14821.380483] LNet: Added LNI 192.168.201.113@tcp118 [8/256/0/180] [14822.536843] LNet: Added LNI 192.168.201.113@tcp119 [8/256/0/180] [14823.447411] LNet: Added LNI 192.168.201.113@tcp120 [8/256/0/180] [14824.422908] LNet: Added LNI 192.168.201.113@tcp121 [8/256/0/180] [14825.311256] LNet: Added LNI 192.168.201.113@tcp122 [8/256/0/180] [14826.197664] LNet: Added LNI 192.168.201.113@tcp123 [8/256/0/180] [14826.954497] LNet: Added LNI 192.168.201.113@tcp124 [8/256/0/180] [14827.874386] LNet: Added LNI 192.168.201.113@tcp125 [8/256/0/180] [14828.745725] LNet: Added LNI 192.168.201.113@tcp126 [8/256/0/180] [14829.810942] LNet: Added LNI 192.168.201.113@tcp127 [8/256/0/180] [14830.851047] LNet: Added LNI 192.168.201.113@tcp128 [8/256/0/180] [14831.666518] LNet: Added LNI 192.168.201.113@tcp129 [8/256/0/180] [14832.764828] LNet: Added LNI 192.168.201.113@tcp130 [8/256/0/180] [14833.918485] LNet: Added LNI 192.168.201.113@tcp131 [8/256/0/180] [14835.054533] LNet: Added LNI 192.168.201.113@tcp132 [8/256/0/180] [14836.219879] LNet: Added LNI 192.168.201.113@tcp133 [8/256/0/180] [14837.268411] LNet: Added LNI 192.168.201.113@tcp134 [8/256/0/180] [14838.612878] LNet: Added LNI 192.168.201.113@tcp135 [8/256/0/180] [14848.300761] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14848.328498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14849.467119] Lustre: 464493:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.113@tcp: can't add NID 192.168.201.113@tcp130: rc = -75 [14849.595509] LNetError: 464521:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.113 [14849.613721] LNetError: Protocol error connecting to 192.168.201.113@tcp117 at host 192.168.201.113:988: is it running a compatible version of Lustre? [14849.660253] LNetError: Skipped 12 previous similar messages [14849.768946] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14849.814668] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14849.960550] Lustre: lustre-MDT0000: new disk, initializing [14850.086218] Lustre: 464551:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.113@tcp: can't add NID 192.168.201.113@tcp130: rc = -75 [14850.094610] Lustre: 464551:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14850.122349] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14850.143274] LNet: 8 local NIs in recovery (showing 8): 192.168.201.113@tcp120, 192.168.201.113@tcp121, 192.168.201.113@tcp122, 192.168.201.113@tcp124, 192.168.201.113@tcp125, 192.168.201.113@tcp123, 192.168.201.113@tcp127, 192.168.201.113@tcp129 [14850.148200] LNetError: 461954:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.113 [14850.207323] LNetError: 461954:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 12 previous similar messages [14850.234951] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14850.235162] LNetError: Protocol error connecting to 192.168.201.113@tcp112 at host 192.168.201.113:988: is it running a compatible version of Lustre? [14857.629413] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [14870.562272] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14870.762834] Lustre: 464512:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14870.863550] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14870.872458] Lustre: Skipped 1 previous similar message [14870.965302] Lustre: lustre-MDT0001: new disk, initializing [14871.036141] Lustre: 465314:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.113@tcp: can't add NID 192.168.201.113@tcp130: rc = -75 [14871.044991] Lustre: 465314:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14871.175204] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14871.296478] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14871.324482] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14876.483321] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [14879.840595] LNetError: 464526:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.113 [14879.859469] LNetError: Protocol error connecting to 192.168.201.113@tcp118 at host 192.168.201.113:988: is it running a compatible version of Lustre? [14880.310894] Lustre: 465696:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.113@tcp: can't add NID 192.168.201.113@tcp130: rc = -75 [14880.340897] Lustre: 465696:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14881.888809] LNetError: 464521:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.113 [14881.898497] LNetError: 464521:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 1 previous similar message [14881.905388] LNetError: Protocol error connecting to 192.168.201.113@tcp113 at host 192.168.201.113:988: is it running a compatible version of Lustre? [14881.926165] LNetError: Skipped 1 previous similar message [14887.023824] LNetError: 464518:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.113 [14887.041543] LNetError: Protocol error connecting to 192.168.201.113@tcp117 at host 192.168.201.113:988: is it running a compatible version of Lustre? [14887.592066] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14894.455806] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14900.325698] LNetError: 464523:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.113 [14900.342937] LNetError: 464523:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 5 previous similar messages [14900.349724] LNetError: Protocol error connecting to 192.168.201.113@tcp117 at host 192.168.201.113:988: is it running a compatible version of Lustre? [14900.365788] LNetError: Skipped 5 previous similar messages [14901.770668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14902.124988] Lustre: lustre-OST0000: new disk, initializing [14902.128704] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14902.139655] Lustre: 466487:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14902.248647] Lustre: 466492:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.113@tcp: can't add NID 192.168.201.113@tcp130: rc = -75 [14902.263643] Lustre: 466492:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14902.312974] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14908.693981] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [14910.979681] Lustre: 466943:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.113@tcp: can't add NID 192.168.201.113@tcp130: rc = -75 [14910.984500] Lustre: 466943:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14911.022980] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14911.036102] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14911.175342] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14917.728448] LNetError: 464519:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.113 [14917.744314] LNetError: 464519:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 7 previous similar messages [14917.758257] LNetError: Protocol error connecting to 192.168.201.113@tcp128 at host 192.168.201.113:988: is it running a compatible version of Lustre? [14917.770637] LNetError: Skipped 7 previous similar messages [14919.377037] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14931.937757] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14931.940084] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14931.949897] LustreError: Skipped 1 previous similar message [14931.951953] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14931.995730] Lustre: Skipped 1 previous similar message [14935.158893] Lustre: server umount lustre-OST0000 complete [14942.691671] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14942.711364] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14942.728557] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14942.731398] Lustre: Skipped 1 previous similar message [14945.249765] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14945.250557] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14945.281907] Lustre: Skipped 1 previous similar message [14947.033320] Lustre: server umount lustre-MDT0000 complete [14951.521132] LNetError: 464515:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.113 [14951.538141] LNetError: 464515:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 9 previous similar messages [14951.544510] LNetError: Protocol error connecting to 192.168.201.113@tcp117 at host 192.168.201.113:988: is it running a compatible version of Lustre? [14951.550302] LNetError: Skipped 9 previous similar messages [14951.897503] LustreError: 464503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786826937 with bad export cookie 14795233413307955583 [14951.911858] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14951.916335] LustreError: 464503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14952.240724] Lustre: server umount lustre-MDT0001 complete [14967.089799] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [14970.584473] Key type lgssc unregistered [14971.147585] LNet: 468403:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14971.189404] LNetError: 468403:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14971.200076] LNet: Removed LNI 192.168.201.113@tcp [14972.255166] LNet: Removed LNI 192.168.201.113@tcp100 [14972.257314] LNetError: 468403:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14972.265978] LNetError: 468403:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14972.282434] LNet: Removed LNI 192.168.201.113@tcp101 [14973.343633] LNet: Removed LNI 192.168.201.113@tcp102 [14973.348433] LNetError: 468403:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14973.365620] LNetError: 468403:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14973.382916] LNet: Removed LNI 192.168.201.113@tcp103 [14974.434755] LNet: Removed LNI 192.168.201.113@tcp104 [14975.455220] LNet: Removed LNI 192.168.201.113@tcp105 [14975.460309] LNetError: 468403:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14975.466689] LNetError: 468403:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14975.472146] LNet: Removed LNI 192.168.201.113@tcp106 [14976.479366] LNet: Removed LNI 192.168.201.113@tcp107 [14977.505362] LNet: Removed LNI 192.168.201.113@tcp108 [14978.527248] LNet: Removed LNI 192.168.201.113@tcp109 [14979.551639] LNet: Removed LNI 192.168.201.113@tcp110 [14979.571636] LNetError: 468403:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14979.581037] LNetError: 468403:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14979.591608] LNet: Removed LNI 192.168.201.113@tcp111 [14980.639205] LNet: Removed LNI 192.168.201.113@tcp112 [14981.663268] LNet: Removed LNI 192.168.201.113@tcp113 [14982.690723] LNet: Removed LNI 192.168.201.113@tcp114 [14983.718508] LNet: Removed LNI 192.168.201.113@tcp115 [14984.735946] LNet: Removed LNI 192.168.201.113@tcp116 [14985.761599] LNet: Removed LNI 192.168.201.113@tcp117 [14986.783757] LNet: Removed LNI 192.168.201.113@tcp118 [14987.807408] LNet: Removed LNI 192.168.201.113@tcp119 [14987.812425] LNetError: 468403:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14987.825881] LNetError: 468403:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [14987.835546] LNet: Removed LNI 192.168.201.113@tcp120 [14988.896079] LNet: Removed LNI 192.168.201.113@tcp121 [14989.983178] LNet: Removed LNI 192.168.201.113@tcp122 [14991.007228] LNet: Removed LNI 192.168.201.113@tcp123 [14992.031167] LNet: Removed LNI 192.168.201.113@tcp124 [14993.055227] LNet: Removed LNI 192.168.201.113@tcp125 [14994.081618] LNet: Removed LNI 192.168.201.113@tcp126 [14995.103752] LNet: Removed LNI 192.168.201.113@tcp127 [14996.127193] LNet: Removed LNI 192.168.201.113@tcp128 [14997.151207] LNet: Removed LNI 192.168.201.113@tcp129 [14997.160069] LNet: Removed LNI 192.168.201.113@tcp130 [14997.167406] LNet: Removed LNI 192.168.201.113@tcp131 [14997.173383] LNet: Removed LNI 192.168.201.113@tcp132 [14997.176921] LNet: Removed LNI 192.168.201.113@tcp133 [14997.178941] LNet: Removed LNI 192.168.201.113@tcp134 [14997.193879] LNet: Removed LNI 192.168.201.113@tcp135 [14998.349884] Key type .llcrypt unregistered [14998.351494] Key type ._llcrypt unregistered [15028.322334] Key type ._llcrypt registered [15028.328352] Key type .llcrypt registered [15028.576187] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [15051.981711] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [15053.459790] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15053.694216] alg: No test for adler32 (adler32-zlib) [15054.954271] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [15055.198227] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [15056.913475] Key type lgssc registered [15058.612957] Lustre: Echo OBD driver; http://www.lustre.org/ [15105.788058] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 16:51:30 (1786827090) [15126.986825] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [15130.236378] Key type lgssc unregistered [15130.631111] LNet: 472500:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15130.634953] LNetError: 472500:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15130.663215] LNet: Removed LNI 192.168.201.113@tcp [15131.677885] Key type .llcrypt unregistered [15131.683852] Key type ._llcrypt unregistered [15154.453591] Key type ._llcrypt registered [15154.455190] Key type .llcrypt registered [15154.612412] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [15155.924625] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15156.029509] alg: No test for adler32 (adler32-zlib) [15157.088962] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [15157.248742] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [15158.879162] Key type lgssc registered [15160.034819] Lustre: Echo OBD driver; http://www.lustre.org/ [15171.325437] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15171.368280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15172.770522] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15172.824864] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15172.942761] Lustre: lustre-MDT0000: new disk, initializing [15173.088863] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15173.145048] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15179.199688] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [15190.947345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15191.063413] Lustre: 473717:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15191.138568] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15191.146816] Lustre: Skipped 1 previous similar message [15191.268432] Lustre: lustre-MDT0001: new disk, initializing [15191.384584] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15191.448257] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15191.456534] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15197.011434] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [15207.194694] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15213.058313] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15220.675377] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15221.084950] Lustre: lustre-OST0000: new disk, initializing [15221.093316] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15221.103565] Lustre: 475683:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15221.254385] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15229.024351] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15229.044562] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15229.180791] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [15229.272902] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15240.722464] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15245.123161] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing lnet_if_list [15250.526938] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15250.561635] LNet: Added LNI 192.168.201.113@tcp10 [8/256/0/180] [15258.065501] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15258.103957] LNet: Added LNI 192.168.201.113@tcp11 [8/256/0/180] [15263.416326] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15263.464355] LNet: Added LNI 192.168.201.113@tcp12 [8/256/0/180] [15269.591848] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15269.651518] LNet: Added LNI 192.168.201.113@tcp13 [8/256/0/180] [15275.078658] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15275.119172] LNet: Added LNI 192.168.201.113@tcp14 [8/256/0/180] [15280.284499] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15280.321546] LNet: Added LNI 192.168.201.113@tcp15 [8/256/0/180] [15286.634948] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15286.656505] LNet: Added LNI 192.168.201.113@tcp16 [8/256/0/180] [15291.590561] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15291.647524] LNet: Added LNI 192.168.201.113@tcp17 [8/256/0/180] [15297.917482] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15297.950498] LNet: Added LNI 192.168.201.113@tcp18 [8/256/0/180] [15305.040105] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15305.088365] LNet: Added LNI 192.168.201.113@tcp19 [8/256/0/180] [15311.212507] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15311.282403] LNet: Added LNI 192.168.201.113@tcp20 [8/256/0/180] [15318.050082] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15318.097475] LNet: Added LNI 192.168.201.113@tcp21 [8/256/0/180] [15324.274715] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15324.307210] LNet: Added LNI 192.168.201.113@tcp22 [8/256/0/180] [15330.771609] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15330.798575] LNet: Added LNI 192.168.201.113@tcp23 [8/256/0/180] [15337.478962] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15337.524787] LNet: Added LNI 192.168.201.113@tcp24 [8/256/0/180] [15345.957983] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15346.008585] LNet: Added LNI 192.168.201.113@tcp25 [8/256/0/180] [15353.823593] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15353.888676] LNet: Added LNI 192.168.201.113@tcp26 [8/256/0/180] [15361.594418] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15361.645912] LNet: Added LNI 192.168.201.113@tcp27 [8/256/0/180] [15369.999733] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15370.043053] LNet: Added LNI 192.168.201.113@tcp28 [8/256/0/180] [15377.263741] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15377.324401] LNet: Added LNI 192.168.201.113@tcp29 [8/256/0/180] [15384.851112] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15384.887423] LNet: Added LNI 192.168.201.113@tcp30 [8/256/0/180] [15392.235743] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15392.276459] LNet: Added LNI 192.168.201.113@tcp31 [8/256/0/180] [15401.013539] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15401.061074] LNet: Added LNI 192.168.201.113@tcp32 [8/256/0/180] [15411.354128] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15411.402709] LNet: Added LNI 192.168.201.113@tcp33 [8/256/0/180] [15422.826231] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15422.845208] LNet: Added LNI 192.168.201.113@tcp34 [8/256/0/180] [15433.294367] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15433.342552] LNet: Added LNI 192.168.201.113@tcp35 [8/256/0/180] [15443.361492] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [15443.380777] LNet: Added LNI 192.168.201.113@tcp36 [8/256/0/180] [15452.464941] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15452.528683] LNet: Added LNI 192.168.201.113@tcp37 [8/256/0/180] [15461.580376] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [15461.634480] LNet: Added LNI 192.168.201.113@tcp38 [8/256/0/180] [15471.967239] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15472.002576] LNet: Added LNI 192.168.201.113@tcp39 [8/256/0/180] [15479.681104] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [15479.721744] LNet: Added LNI 192.168.201.113@tcp40 [8/256/0/180] [15490.983496] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [15491.010958] LNet: Added LNI 192.168.201.113@tcp41 [8/256/0/180] [15501.719939] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15501.773558] LNet: Added LNI 192.168.201.113@tcp42 [8/256/0/180] [15511.247584] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15511.315758] LNet: Added LNI 192.168.201.113@tcp43 [8/256/0/180] [15521.385160] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15521.438701] LNet: Added LNI 192.168.201.113@tcp44 [8/256/0/180] [15531.738601] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15531.755651] LNet: Added LNI 192.168.201.113@tcp45 [8/256/0/180] [15542.185409] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15542.237142] LNet: Added LNI 192.168.201.113@tcp46 [8/256/0/180] [15551.253698] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15551.288358] LNet: Added LNI 192.168.201.113@tcp47 [8/256/0/180] [15560.380989] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15560.425214] LNet: Added LNI 192.168.201.113@tcp48 [8/256/0/180] [15570.169838] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15570.188483] LNet: Added LNI 192.168.201.113@tcp49 [8/256/0/180] [15580.031897] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15580.055443] LNet: Added LNI 192.168.201.113@tcp50 [8/256/0/180] [15603.169514] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15603.173713] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15603.189075] Lustre: Skipped 2 previous similar messages [15603.194977] Lustre: Skipped 1 previous similar message [15608.239528] Lustre: server umount lustre-MDT0000 complete [15612.884282] LustreError: 482412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786827598 with bad export cookie 9392487070770006150 [15612.884751] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15612.892102] LustreError: 482412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15613.375032] Lustre: server umount lustre-MDT0001 complete [15629.797301] Lustre: 472987:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786827599/real 1786827599] req@0000000052f19505 x1873624058493696/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786827615 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15629.828796] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15633.512022] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [15633.571298] Lustre: 472988:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786827603/real 1786827603] req@0000000027147b1f x1873624058493824/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786827619 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15634.916805] Lustre: 472990:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786827604/real 1786827604] req@00000000ac1129d1 x1873624058493952/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786827620 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15639.009187] Lustre: 472989:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786827608/real 1786827608] req@0000000007a29c5d x1873624058494208/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786827624 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15643.035598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15643.108337] Lustre: 472987:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786827613/real 1786827613] req@000000002326dcd3 x1873624058494336/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786827629 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15657.567491] LustreError: 472986:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000a1bb0c83 x1873624058495104/t0(0) o250->MGC192.168.201.113@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15657.875363] LustreError: 475677:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15657.902132] LustreError: 475677:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [15657.925783] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15658.930435] LustreError: 475678:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15658.936070] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15659.940808] LustreError: 475677:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15663.136000] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [15665.139302] LustreError: 475678:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15670.241441] LustreError: 475677:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15673.849506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15674.473648] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15675.499665] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15682.513105] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [15693.102994] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15701.881723] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15729.633427] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15729.638845] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15729.655630] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15729.657681] Lustre: Skipped 1 previous similar message [15730.657520] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15730.673532] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15735.921594] Lustre: server umount lustre-OST0000 complete [15740.898553] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15740.911913] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15740.934340] Lustre: Skipped 1 previous similar message [15740.941855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15740.947480] Lustre: Skipped 2 previous similar messages [15746.886305] Lustre: server umount lustre-MDT0000 complete [15747.555665] LustreError: 499938:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15747.579412] LustreError: 499938:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [15751.145000] LustreError: 482412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786827737 with bad export cookie 9392487070770008782 [15751.148909] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15751.157053] LustreError: 482412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15751.619246] Lustre: server umount lustre-MDT0001 complete [15769.358123] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [15773.123464] Key type lgssc unregistered [15773.613075] LNet: 502778:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15773.622214] LNetError: 502778:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15773.633786] LNet: Removed LNI 192.168.201.113@tcp [15773.639898] LNet: Removed LNI 192.168.201.113@tcp10 [15773.648934] LNet: Removed LNI 192.168.201.113@tcp11 [15773.651096] LNet: Removed LNI 192.168.201.113@tcp12 [15773.664721] LNet: Removed LNI 192.168.201.113@tcp13 [15773.666419] LNet: Removed LNI 192.168.201.113@tcp14 [15773.674777] LNet: Removed LNI 192.168.201.113@tcp15 [15773.681592] LNet: Removed LNI 192.168.201.113@tcp16 [15773.690114] LNet: Removed LNI 192.168.201.113@tcp17 [15773.696521] LNet: Removed LNI 192.168.201.113@tcp18 [15773.705936] LNet: Removed LNI 192.168.201.113@tcp19 [15773.721031] LNet: Removed LNI 192.168.201.113@tcp20 [15773.733888] LNet: Removed LNI 192.168.201.113@tcp21 [15773.736843] LNet: Removed LNI 192.168.201.113@tcp22 [15773.739573] LNet: Removed LNI 192.168.201.113@tcp23 [15773.742336] LNet: Removed LNI 192.168.201.113@tcp24 [15773.747336] LNet: Removed LNI 192.168.201.113@tcp25 [15773.750196] LNet: Removed LNI 192.168.201.113@tcp26 [15773.756227] LNet: Removed LNI 192.168.201.113@tcp27 [15773.758662] LNet: Removed LNI 192.168.201.113@tcp28 [15773.760515] LNet: Removed LNI 192.168.201.113@tcp29 [15773.764789] LNet: Removed LNI 192.168.201.113@tcp30 [15773.767961] LNet: Removed LNI 192.168.201.113@tcp31 [15773.777954] LNet: Removed LNI 192.168.201.113@tcp32 [15773.780906] LNet: Removed LNI 192.168.201.113@tcp33 [15773.787229] LNet: Removed LNI 192.168.201.113@tcp34 [15773.789489] LNet: Removed LNI 192.168.201.113@tcp35 [15773.792929] LNet: Removed LNI 192.168.201.113@tcp36 [15773.797673] LNet: Removed LNI 192.168.201.113@tcp37 [15773.804608] LNet: Removed LNI 192.168.201.113@tcp38 [15773.810728] LNet: Removed LNI 192.168.201.113@tcp39 [15773.817822] LNet: Removed LNI 192.168.201.113@tcp40 [15773.823651] LNet: Removed LNI 192.168.201.113@tcp41 [15773.825346] LNet: Removed LNI 192.168.201.113@tcp42 [15773.827737] LNet: Removed LNI 192.168.201.113@tcp43 [15773.830544] LNet: Removed LNI 192.168.201.113@tcp44 [15773.837609] LNet: Removed LNI 192.168.201.113@tcp45 [15773.840406] LNet: Removed LNI 192.168.201.113@tcp46 [15773.843218] LNet: Removed LNI 192.168.201.113@tcp47 [15773.845834] LNet: Removed LNI 192.168.201.113@tcp48 [15773.848574] LNet: Removed LNI 192.168.201.113@tcp49 [15773.878695] LNet: Removed LNI 192.168.201.113@tcp50 [15774.943655] Key type .llcrypt unregistered [15774.945356] Key type ._llcrypt unregistered [15796.676682] Key type ._llcrypt registered [15796.680560] Key type .llcrypt registered [15796.759731] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [15813.161300] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [15814.442414] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15814.569309] alg: No test for adler32 (adler32-zlib) [15815.638546] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [15815.924248] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [15817.696390] Key type lgssc registered [15818.937670] Lustre: Echo OBD driver; http://www.lustre.org/ [15859.939253] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15860.005755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15861.296990] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15861.328311] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15861.425197] Lustre: lustre-MDT0000: new disk, initializing [15861.551655] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15861.583913] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15867.005588] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [15879.525583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15879.763327] Lustre: 505979:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15879.799681] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15879.802221] Lustre: Skipped 1 previous similar message [15879.854187] Lustre: lustre-MDT0001: new disk, initializing [15879.954108] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15880.005495] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15880.017338] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15885.569901] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [15896.842449] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15903.441602] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15913.218264] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15913.476684] Lustre: *** cfs_fail_loc=90f, val=0*** [15913.561540] Lustre: lustre-OST0000: new disk, initializing [15913.572636] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15913.583506] Lustre: 507996:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15913.743590] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15920.205327] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15920.226682] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15920.361047] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15921.021845] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [15932.000027] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15941.091246] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15941.093333] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15941.101816] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15941.110790] Lustre: Skipped 1 previous similar message [15946.314148] Lustre: server umount lustre-OST0000 complete [15951.332676] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15951.342327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15951.343170] Lustre: Skipped 1 previous similar message [15951.357920] Lustre: Skipped 2 previous similar messages [15956.452827] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15956.457743] Lustre: Skipped 1 previous similar message [15956.947752] Lustre: server umount lustre-MDT0000 complete [15961.424304] LustreError: 505970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786827947 with bad export cookie 14435126670336839106 [15961.434453] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15961.438693] LustreError: 505970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [15961.980113] Lustre: server umount lustre-MDT0001 complete [15975.889081] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [15979.273164] Key type lgssc unregistered [15979.739500] LNet: 509904:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15979.744967] LNetError: 509904:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15979.775163] LNet: Removed LNI 192.168.201.113@tcp [15980.932767] Key type .llcrypt unregistered [15980.935922] Key type ._llcrypt unregistered [16004.543412] Key type ._llcrypt registered [16004.547380] Key type .llcrypt registered [16004.855991] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [16006.385855] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16006.540045] alg: No test for adler32 (adler32-zlib) [16007.705409] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [16007.968834] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [16009.687306] Key type lgssc registered [16010.838433] Lustre: Echo OBD driver; http://www.lustre.org/ [16019.394368] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16019.418328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16021.059656] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16026.288198] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [16030.180945] LustreError: 511177:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16030.202040] LustreError: 511177:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [16035.098893] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16035.298439] LustreError: 511178:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16035.677611] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16040.852767] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [16049.313372] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16057.295946] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16066.061071] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16066.802592] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16073.438938] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [16083.269123] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16106.377512] loop0: detected capacity change from 0 to 204800000 [16106.695873] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16106.726196] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16106.760762] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16115.553841] loop0: detected capacity change from 0 to 204800000 [16115.784154] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16115.809245] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16115.826075] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16121.155767] loop0: detected capacity change from 0 to 204800000 [16127.081581] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16127.229374] Lustre: fs15246-OST0000: new disk, initializing [16127.238730] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [16127.246797] Lustre: 514799:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16127.343829] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16134.548849] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [16145.786502] loop1: detected capacity change from 0 to 204800000 [16152.149458] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16152.295605] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [16152.363418] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [16152.442839] Lustre: fs15246-MDT0000: new disk, initializing [16152.594840] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16152.617415] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16152.735058] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [16153.760123] Lustre: 515921:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000007edef827 x1873624949640320/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786828155 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [16157.609431] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [16162.852878] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [16162.870068] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [16162.900416] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [16182.756574] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16182.781788] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16186.703674] Lustre: server umount fs15246-MDT0000 complete [16187.877351] LustreError: 511174:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16193.005166] LustreError: 513680:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16196.273765] Lustre: server umount fs15246-OST0000 complete [16208.357946] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16208.370778] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16208.377188] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16208.387894] Lustre: Skipped 1 previous similar message [16208.395956] LustreError: Skipped 1 previous similar message [16213.478131] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16213.487332] Lustre: Skipped 2 previous similar messages [16218.601326] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16218.606199] Lustre: Skipped 1 previous similar message [16221.151891] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16221.502569] Lustre: server umount lustre-OST0000 complete [16228.834969] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16228.837079] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16228.868319] Lustre: Skipped 1 previous similar message [16228.880484] Lustre: Skipped 1 previous similar message [16232.581058] Lustre: server umount lustre-MDT0000 complete [16233.952774] LustreError: 511173:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16233.979748] LustreError: 511173:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16236.943889] LustreError: 511157:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786828222 with bad export cookie 9886476326284248414 [16236.963443] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16237.444327] Lustre: server umount lustre-MDT0001 complete [16249.205504] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [16252.280805] Key type lgssc unregistered [16252.482790] LNet: 519021:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16252.485988] LNetError: 519021:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16252.502906] LNet: Removed LNI 192.168.201.113@tcp [16253.493901] Key type .llcrypt unregistered [16253.496113] Key type ._llcrypt unregistered [16273.278955] Key type ._llcrypt registered [16273.280790] Key type .llcrypt registered [16273.393118] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [16274.495620] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16274.550814] alg: No test for adler32 (adler32-zlib) [16275.576754] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [16275.792945] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [16277.479229] Key type lgssc registered [16278.675207] Lustre: Echo OBD driver; http://www.lustre.org/ [16320.727746] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [16331.717367] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [16378.976707] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16378.987287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16380.292758] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16380.308448] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16380.362650] Lustre: lustre-MDT0000: new disk, initializing [16380.570059] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16380.595990] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16385.812302] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [16398.487766] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16398.662037] Lustre: 523491:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16398.730215] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16398.733143] Lustre: Skipped 1 previous similar message [16398.838440] Lustre: lustre-MDT0001: new disk, initializing [16398.948132] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16398.978469] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16398.997776] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16403.885233] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [16413.362388] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16419.780152] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16428.188777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16428.752557] Lustre: lustre-OST0000: new disk, initializing [16428.756609] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16428.763734] Lustre: 525465:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16428.852634] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16435.298553] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [16437.823982] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16437.833978] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16437.943761] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16446.288285] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16451.633066] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16451.850262] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16456.951195] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16457.114978] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16463.841487] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16463.848026] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16463.850123] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16463.860590] Lustre: Skipped 1 previous similar message [16463.872613] Lustre: Skipped 1 previous similar message [16465.344661] Lustre: server umount lustre-OST0000 complete [16470.498350] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16470.513235] LustreError: Skipped 1 previous similar message [16470.522761] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16470.543828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16474.082252] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16474.087679] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16475.859434] Lustre: server umount lustre-MDT0000 complete [16479.862840] LustreError: 523482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786828465 with bad export cookie 4206942181456613425 [16479.873244] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16479.878560] LustreError: 523482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16480.350895] Lustre: server umount lustre-MDT0001 complete [16505.985679] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [16516.458987] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [16557.361618] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16234 sec ========= 17:15:41 (1786828541) [16560.003304] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 17:15:44 (1786828544) === [16565.489648] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 17:15:49 (1786828549) === [16608.751666] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [16611.442905] Key type lgssc unregistered [16611.698560] LNet: 532112:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16611.710102] LNetError: 532112:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16611.723666] LNet: Removed LNI 192.168.201.113@tcp [16612.489568] Key type .llcrypt unregistered [16612.492431] Key type ._llcrypt unregistered