[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 467421738 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001017] APIC: Switch to symmetric I/O mode setup [ 0.003238] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005017] kvm-guest: setup PV IPIs [ 0.008447] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009028] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010015] pid_max: default: 32768 minimum: 301 [ 0.011160] LSM: Security Framework initializing [ 0.012055] Yama: becoming mindful. [ 0.013032] SELinux: Initializing. [ 0.015028] *** VALIDATE selinux *** [ 0.023714] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028232] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029193] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030120] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031125] *** VALIDATE tmpfs *** [ 0.033497] *** VALIDATE proc *** [ 0.035071] *** VALIDATE cgroup *** [ 0.036009] *** VALIDATE cgroup2 *** [ 0.037284] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038166] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040037] Spectre V2 : User space: Vulnerable [ 0.041013] Speculative Store Bypass: Vulnerable [ 0.044682] debug: unmapping init [mem 0xffffffff8a859000-0xffffffff8a860fff] [ 0.046217] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047761] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048023] ... version: 2 [ 0.048808] ... bit width: 48 [ 0.049011] ... generic registers: 4 [ 0.050010] ... value mask: 0000ffffffffffff [ 0.051014] ... max period: 00007fffffffffff [ 0.052027] ... fixed-purpose events: 3 [ 0.053011] ... event mask: 000000070000000f [ 0.054368] rcu: Hierarchical SRCU implementation. [ 0.056633] smp: Bringing up secondary CPUs ... [ 0.057623] x86: Booting SMP configuration: [ 0.058032] .... node #0, CPUs: #1 #2 #3 [ 0.061308] smp: Brought up 1 node, 4 CPUs [ 0.062845] smpboot: Max logical packages: 1 [ 0.063016] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.256894] node 0 deferred pages initialised in 191ms [ 0.258131] devtmpfs: initialized [ 0.259260] x86/mm: Memory block size: 128MB [ 0.262844] gcov: version magic: 0x41383552 [ 0.266105] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.267105] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.268311] pinctrl core: initialized pinctrl subsystem [ 0.269396] [ 0.270022] ************************************************************* [ 0.272023] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.274026] ** ** [ 0.277021] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.279022] ** ** [ 0.281025] ** This means that this kernel is built to expose internal ** [ 0.283021] ** IOMMU data structures, which may compromise security on ** [ 0.286024] ** your system. ** [ 0.288021] ** ** [ 0.290020] ** If you see this message and you are not debugging the ** [ 0.292023] ** kernel, report this immediately to your vendor! ** [ 0.294020] ** ** [ 0.296017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.297013] ************************************************************* [ 0.299894] NET: Registered protocol family 16 [ 0.302600] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.305091] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.307053] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.311046] cpuidle: using governor menu [ 0.312796] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.315400] PCI: Using configuration type 1 for base access [ 0.316115] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.323071] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.324016] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.325228] cryptd: max_cpu_qlen set to 1000 [ 0.328121] ACPI: Added _OSI(Module Device) [ 0.329024] ACPI: Added _OSI(Processor Device) [ 0.331030] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.333023] ACPI: Added _OSI(Processor Aggregator Device) [ 0.338000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.346255] ACPI: Interpreter enabled [ 0.347046] ACPI: PM: (supports S0 S3 S4 S5) [ 0.348014] ACPI: Using IOAPIC for interrupt routing [ 0.350137] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.355374] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.365038] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.367026] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.368015] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.371095] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.376418] acpiphp: Slot [2] registered [ 0.378153] acpiphp: Slot [5] registered [ 0.379098] acpiphp: Slot [6] registered [ 0.380017] acpiphp: Slot [7] registered [ 0.380928] acpiphp: Slot [8] registered [ 0.381078] acpiphp: Slot [9] registered [ 0.381829] acpiphp: Slot [10] registered [ 0.383064] acpiphp: Slot [3] registered [ 0.383906] acpiphp: Slot [4] registered [ 0.385058] acpiphp: Slot [11] registered [ 0.385833] acpiphp: Slot [12] registered [ 0.386059] acpiphp: Slot [13] registered [ 0.386965] acpiphp: Slot [14] registered [ 0.388053] acpiphp: Slot [15] registered [ 0.388863] acpiphp: Slot [16] registered [ 0.389049] acpiphp: Slot [17] registered [ 0.389893] acpiphp: Slot [18] registered [ 0.392114] acpiphp: Slot [19] registered [ 0.393084] acpiphp: Slot [20] registered [ 0.394102] acpiphp: Slot [21] registered [ 0.395066] acpiphp: Slot [22] registered [ 0.395994] acpiphp: Slot [23] registered [ 0.397074] acpiphp: Slot [24] registered [ 0.397960] acpiphp: Slot [25] registered [ 0.399073] acpiphp: Slot [26] registered [ 0.400117] acpiphp: Slot [27] registered [ 0.401063] acpiphp: Slot [28] registered [ 0.402071] acpiphp: Slot [29] registered [ 0.403125] acpiphp: Slot [30] registered [ 0.404086] acpiphp: Slot [31] registered [ 0.405063] PCI host bridge to bus 0000:00 [ 0.407021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.409035] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.412037] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.414030] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.416033] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.419051] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.421276] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.424195] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.426012] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.436633] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.441050] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.443022] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.445020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.448024] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.450591] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.453915] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.456052] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.458538] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.463016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.476020] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.481020] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.486668] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.496041] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.505022] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.517020] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.525911] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.535021] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.546024] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.563019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.570675] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.578015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.583019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.598025] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.606126] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.611014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.616020] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.633024] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.642000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.646014] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.650017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.664017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.673481] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.679015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.683019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.696027] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.706864] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.709461] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.711349] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.713501] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.715241] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.720465] iommu: Default domain type: Passthrough [ 0.723704] SCSI subsystem initialized [ 0.725233] ACPI: bus type USB registered [ 0.727225] usbcore: registered new interface driver usbfs [ 0.729182] usbcore: registered new interface driver hub [ 0.731178] usbcore: registered new device driver usb [ 0.734262] pps_core: LinuxPPS API ver. 1 registered [ 0.736014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.739104] PTP clock support registered [ 0.742066] EDAC MC: Ver: 3.0.0 [ 0.744091] PCI: Using ACPI for IRQ routing [ 0.746040] NetLabel: Initializing [ 0.747011] NetLabel: domain hash size = 128 [ 0.749012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.750104] NetLabel: unlabeled traffic allowed by default [ 0.753172] vgaarb: loaded [ 0.754302] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.756021] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.765000] clocksource: Switched to clocksource kvm-clock [ 0.880682] VFS: Disk quotas dquot_6.6.0 [ 0.882439] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.884558] *** VALIDATE ramfs *** [ 0.885741] *** VALIDATE hugetlbfs *** [ 0.887174] pnp: PnP ACPI init [ 0.889616] pnp: PnP ACPI: found 6 devices [ 0.905888] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.908201] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.909729] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.911819] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.914194] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.916324] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.918806] NET: Registered protocol family 2 [ 0.921353] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.926450] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.928875] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.934500] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.938134] TCP: Hash tables configured (established 65536 bind 65536) [ 0.941180] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.944439] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.947329] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.950181] NET: Registered protocol family 1 [ 0.952786] RPC: Registered named UNIX socket transport module. [ 0.955215] RPC: Registered udp transport module. [ 0.957123] RPC: Registered tcp transport module. [ 0.958631] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.961248] NET: Registered protocol family 44 [ 0.963441] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.965813] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.968376] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.970786] PCI: CLS 0 bytes, default 64 [ 0.972426] Unpacking initramfs... [ 2.379495] debug: unmapping init [mem 0xffff8d4d7cc54000-0xffff8d4d7ffbffff] [ 2.384310] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.386523] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.389469] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.895593] Initialise system trusted keyrings [ 2.897482] Key type blacklist registered [ 2.899867] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.911975] zbud: loaded [ 2.915918] *** VALIDATE nfs *** [ 2.917264] *** VALIDATE nfs4 *** [ 2.918971] pstore: using deflate compression [ 2.925321] Platform Keyring initialized [ 3.040687] NET: Registered protocol family 38 [ 3.042172] Key type asymmetric registered [ 3.043311] Asymmetric key parser 'x509' registered [ 3.044502] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.046653] io scheduler mq-deadline registered [ 3.047683] io scheduler kyber registered [ 3.048905] io scheduler bfq registered [ 3.050382] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.052537] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.054603] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.057167] ACPI: Power Button [PWRF] [ 3.061683] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.068398] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.082650] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.092582] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.111521] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.140258] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.170733] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.176305] Non-volatile memory driver v1.3 [ 3.177914] Linux agpgart interface v0.103 [ 3.211392] virtio_blk virtio1: [vda] 145896 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.214671] vda: detected capacity change from 0 to 74698752 [ 3.230295] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.233056] vdb: detected capacity change from 0 to 1073741824 [ 3.249768] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.252734] vdc: detected capacity change from 0 to 2621440000 [ 3.268321] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.284648] vdd: detected capacity change from 0 to 2621440000 [ 3.315885] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.319323] vde: detected capacity change from 0 to 4294967296 [ 3.339522] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.342644] vdf: detected capacity change from 0 to 4294967296 [ 3.354668] libphy: Fixed MDIO Bus: probed [ 3.371388] usbcore: registered new interface driver usbserial_generic [ 3.373540] usbserial: USB Serial support registered for generic [ 3.375988] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.379380] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.381421] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.383895] mousedev: PS/2 mouse device common for all mice [ 3.386085] rtc_cmos 00:05: RTC can wake from S4 [ 3.388100] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.388877] rtc_cmos 00:05: registered as rtc0 [ 3.393022] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.398328] intel_pstate: CPU model not supported [ 3.399220] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.404956] hid: raw HID events driver (C) Jiri Kosina [ 3.405085] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.407376] usbcore: registered new interface driver usbhid [ 3.411835] usbhid: USB HID core driver [ 3.413759] drop_monitor: Initializing network drop monitor service [ 3.416663] Initializing XFRM netlink socket [ 3.418549] NET: Registered protocol family 10 [ 3.421162] Segment Routing with IPv6 [ 3.422644] NET: Registered protocol family 17 [ 3.424893] mpls_gso: MPLS GSO support [ 3.430488] RAS: Correctable Errors collector initialized. [ 3.432074] AVX version of gcm_enc/dec engaged. [ 3.433386] AES CTR mode by8 optimization enabled [ 3.526467] sched_clock: Marking stable (3526441156, 0)->(4415970111, -889528955) [ 3.529560] registered taskstats version 1 [ 3.531090] Loading compiled-in X.509 certificates [ 3.532864] zswap: loaded using pool lzo/zbud [ 3.556908] Key type big_key registered [ 3.568949] Key type encrypted registered [ 3.570229] ima: No TPM chip found, activating TPM-bypass! [ 3.572038] ima: Allocated hash algorithm: sha1 [ 3.573689] ima: No architecture policies found [ 3.575495] evm: Initialising EVM extended attributes: [ 3.576919] evm: security.selinux [ 3.577896] evm: security.ima [ 3.579145] evm: security.capability [ 3.580043] evm: HMAC attrs: 0x1 [ 3.581840] rtc_cmos 00:05: setting system clock to 2026-09-03 13:10:54 UTC (1788441054) [ 3.587598] debug: unmapping init [mem 0xffffffff8b803000-0xffffffff8b9fffff] [ 3.590896] debug: unmapping init [mem 0xffffffff8a582000-0xffffffff8a858fff] [ 3.600074] Write protecting the kernel read-only data: 28672k [ 3.602248] debug: unmapping init [mem 0xffffffff88c03000-0xffffffff88dfffff] [ 3.604109] debug: unmapping init [mem 0xffffffff89514000-0xffffffff895fffff] [ 3.645606] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.653417] systemd[1]: Detected virtualization kvm. [ 3.655564] systemd[1]: Detected architecture x86-64. [ 3.656843] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.680257] systemd[1]: No hostname configured. [ 3.681890] systemd[1]: Set hostname to . [ 3.683507] random: systemd: uninitialized urandom read (16 bytes read) [ 3.685135] systemd[1]: Initializing machine ID from random generator. [ 3.802108] random: systemd: uninitialized urandom read (16 bytes read) [ 3.804390] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.807700] random: systemd: uninitialized urandom read (16 bytes read) [ 3.809703] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.813972] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Paths. [ OK ] Reached target Slices. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Timers. Starting Setup Virtual Console... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.406597] device-mapper: uevent: version 1.0.3 [ 4.408435] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ 5.088259] virtio_net virtio0 ens2: renamed from eth0 [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.105369] random: fast init done [ 5.159519] scsi host0: ata_piix [ 5.170704] scsi host1: ata_piix [ 5.171826] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.173746] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.437137] dracut-initqueue[583]: RTNETLINK answers: File exists [ 10.091553] random: crng init done [ 10.093019] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.433574] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.484827] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.711648] SELinux: Disabled at runtime. [ 11.773344] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.788081] systemd[1]: Detected virtualization kvm. [ 11.789527] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.246830] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.249714] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.254910] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.259086] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.262505] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.270318] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.276308] systemd[1]: Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. Activating swap /dev/disk/by-label/SWAP... Mounting POSIX Message Queue File System... [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ 12.308919] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Huge Pages File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on RPCbind Server Activation Socket. Mounting Kernel Debug File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. [ OK ] Reached target rpc_pipefs.target. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Reached target RPC Port Mapper. Starting Remount Root and Kernel File Systems... Starting Apply Kernel Variables... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.659448] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.955050] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.978794] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.029249] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.037378] EDAC sbridge: Ver: 1.1.2 [ 14.480476] Key type dns_resolver registered [ 14.778358] NFS: Registering the id_resolver key type [ 14.780867] Key type id_resolver registered [ 14.782641] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg160-server login: [ 45.994190] hrtimer: interrupt took 4365351 ns [ 70.957466] libcfs: loading out-of-tree module taints kernel. [ 71.043889] Key type ._llcrypt registered [ 71.045352] Key type .llcrypt registered [ 71.216148] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_hostid [ 88.613407] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 89.847822] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 89.855402] alg: No test for adler32 (adler32-zlib) [ 91.366360] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 92.338408] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [ 94.127184] Key type lgssc registered [ 95.732728] Lustre: Echo OBD driver; http://www.lustre.org/ [ 112.780076] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 161.273991] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 177.565460] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 177.635521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 178.976195] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 179.007824] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 179.096982] Lustre: lustre-MDT0000: new disk, initializing [ 179.163028] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 179.177847] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 185.853388] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 201.177561] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 201.293136] Lustre: 6523:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 201.323906] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 201.333966] Lustre: Skipped 1 previous similar message [ 201.424336] Lustre: lustre-MDT0001: new disk, initializing [ 201.487284] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 201.519205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 201.535117] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 207.070728] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 212.684879] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 223.443985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 223.641416] Lustre: lustre-OST0000: new disk, initializing [ 223.647400] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 223.659604] Lustre: 8462:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 223.779715] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 223.934766] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 223.944559] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 223.978905] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 230.948864] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 247.282394] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 247.509988] Lustre: lustre-OST0001: new disk, initializing [ 247.512516] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 247.520162] Lustre: 9534:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 247.574124] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 254.020209] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 254.028485] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 254.098109] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 255.289465] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 267.673984] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 274.712242] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 281.294406] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing check_logdir /tmp/testlogs/ [ 288.086527] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing yml_node [ 293.339829] Lustre: DEBUG MARKER: Client: 2.17.56.51 [ 296.403720] Lustre: DEBUG MARKER: MDS: 2.17.56.51 [ 299.281309] Lustre: DEBUG MARKER: OSS: 2.17.56.51 [ 301.234860] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 3 09:15:50 EDT 2026 [ 321.121101] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 322.868904] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 339.936267] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 339.953611] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 339.984153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 341.472601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 341.477299] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 341.488173] Lustre: Skipped 1 previous similar message [ 341.497966] Lustre: Skipped 1 previous similar message [ 344.582657] Lustre: server umount lustre-MDT0000 complete [ 348.366254] LustreError: 6515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441399 with bad export cookie 5066171937602162842 [ 348.367573] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 348.383268] LustreError: 6515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 351.712216] LustreError: 9782:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 351.713758] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 351.716344] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 351.739879] LustreError: 9782:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 351.783436] Lustre: Skipped 2 previous similar messages [ 353.760843] LustreError: 6530:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 353.763947] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 353.786640] LustreError: 6530:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 353.803022] Lustre: Skipped 2 previous similar messages [ 354.727193] Lustre: server umount lustre-MDT0001 complete [ 358.921770] Lustre: server umount lustre-OST0000 complete [ 362.463296] Lustre: server umount lustre-OST0001 complete [ 370.021909] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_hostid [ 381.291763] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 422.116570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 422.423990] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 422.470102] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 422.558807] Lustre: lustre-MDT0000: new disk, initializing [ 422.625160] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 422.658060] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 428.398301] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 443.456288] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 443.748849] Lustre: 15091:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 443.844948] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 443.848929] Lustre: Skipped 1 previous similar message [ 443.979892] Lustre: lustre-MDT0001: new disk, initializing [ 444.176763] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 444.242893] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 444.279899] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 449.863218] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 460.672331] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 467.745631] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 475.693549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 476.059802] Lustre: lustre-OST0000: new disk, initializing [ 476.067762] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 476.074890] Lustre: 17059:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 476.178760] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 477.810158] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 477.821139] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 477.986932] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 483.787329] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 494.057794] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 499.754787] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 500.016978] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 505.346871] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 505.608633] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 508.911694] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 508.926399] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 508.944946] Lustre: Skipped 1 previous similar message [ 508.966572] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 519.155210] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 519.166106] Lustre: Skipped 4 previous similar messages [ 522.207193] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 522.339293] Lustre: server umount lustre-OST0000 complete [ 529.379747] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 529.399264] Lustre: Skipped 1 previous similar message [ 533.539927] Lustre: server umount lustre-MDT0000 complete [ 534.500518] LustreError: 15099:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 534.525453] LustreError: 15099:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 538.009852] LustreError: 15083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441588 with bad export cookie 5066171937602164375 [ 538.011816] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 538.037805] LustreError: 15083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 538.389497] Lustre: server umount lustre-MDT0001 complete [ 545.281866] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 09:19:54 (1788441594) [ 566.301989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 566.652062] LustreError: 19936:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 566.761283] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 571.880390] LustreError: 19937:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 572.038754] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 582.000912] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 582.120849] LustreError: 19936:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 582.135945] LustreError: 19936:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 587.356368] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 595.931134] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 601.334829] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 604.617288] Lustre: Failing over lustre-MDT0000 [ 605.004928] Lustre: server umount lustre-MDT0000 complete [ 607.715143] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 607.726498] LustreError: 19936:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 607.739616] Lustre: Skipped 1 previous similar message [ 624.097477] Lustre: 3657:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788441658/real 1788441658] req@ffff8d4cc315c380 x1875316460129792/t0(0) o400->MGC192.168.201.160@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788441674 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 624.135282] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 626.366354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 635.362823] LustreError: 3654:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8d4dc148ea00 x1875316460132992/t0(0) o250->MGC192.168.201.160@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 635.661625] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 635.669209] Lustre: Skipped 1 previous similar message [ 635.727890] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 640.593317] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 640.997049] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 641.010817] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 641.037216] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 653.658961] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 658.139954] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 658.141396] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 658.155745] Lustre: Skipped 1 previous similar message [ 658.172674] Lustre: Skipped 3 previous similar messages [ 664.768645] Lustre: server umount lustre-MDT0000 complete [ 668.643675] LustreError: 20941:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 668.671153] LustreError: 20941:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 18 previous similar messages [ 669.232492] LustreError: 19917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441720 with bad export cookie 5066171937602165789 [ 669.242686] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 669.825492] Lustre: server umount lustre-MDT0001 complete [ 679.251339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 685.650436] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 694.999211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 700.133608] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 709.200374] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 717.172168] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 719.746025] Lustre: Failing over lustre-MDT0000 [ 720.216957] Lustre: server umount lustre-MDT0000 complete [ 720.804552] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 720.822842] Lustre: Skipped 1 previous similar message [ 736.224513] LustreError: 23272:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 736.246833] LustreError: 23272:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 737.247144] Lustre: 3657:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788441771/real 1788441771] req@ffff8d4cc302f100 x1875316460161792/t0(0) o400->MGC192.168.201.160@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788441787 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 737.277387] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 741.133885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 748.992447] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 754.162715] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 754.164877] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 754.203167] Lustre: Skipped 1 previous similar message [ 754.238297] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 754.566880] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 766.915507] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 774.624865] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 774.657451] LustreError: Skipped 1 previous similar message [ 774.664598] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 774.676182] Lustre: Skipped 2 previous similar messages [ 785.378523] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 785.615524] Lustre: server umount lustre-MDT0000 complete [ 790.582135] LustreError: 23256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441841 with bad export cookie 5066171937602166727 [ 790.587748] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 790.599744] LustreError: 23256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 791.010286] Lustre: server umount lustre-MDT0001 complete [ 802.419113] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 09:24:11 (1788441851) [ 821.259581] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_hostid [ 834.192868] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 878.707783] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 892.218041] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 892.636779] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 892.713256] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 892.872041] Lustre: lustre-MDT0000: new disk, initializing [ 893.077713] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 893.086763] Lustre: Skipped 3 previous similar messages [ 893.135361] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 898.332842] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 910.038349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 910.214971] Lustre: 29658:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 910.689266] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 915.820600] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 926.811755] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 934.613890] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 941.891042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 942.220718] Lustre: 31630:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 944.313954] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 944.386763] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 944.468535] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 944.490017] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 948.875969] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 958.808802] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 966.446856] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 985.571213] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 985.575455] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 985.581127] LustreError: Skipped 1 previous similar message [ 985.593770] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 985.601000] Lustre: Skipped 3 previous similar messages [ 985.619748] Lustre: Skipped 5 previous similar messages [ 990.977882] Lustre: server umount lustre-OST0000 complete [ 1001.279937] Lustre: server umount lustre-MDT0000 complete [ 1005.006506] LustreError: 29649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442055 with bad export cookie 5066171937602167133 [ 1005.010404] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1005.015953] LustreError: 29649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1005.662199] Lustre: server umount lustre-MDT0001 complete [ 1019.377574] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 1023.677551] Key type lgssc unregistered [ 1024.059567] LNet: 33615:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1024.071456] LNetError: 33615:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1024.101643] LNet: Removed LNI 192.168.201.160@tcp [ 1025.390564] Key type .llcrypt unregistered [ 1025.396108] Key type ._llcrypt unregistered [ 1053.730548] Key type ._llcrypt registered [ 1053.733875] Key type .llcrypt registered [ 1053.848253] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 1055.320275] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1055.463641] alg: No test for adler32 (adler32-zlib) [ 1056.457288] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1056.685846] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [ 1058.352758] Key type lgssc registered [ 1059.629318] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1068.697834] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1068.720980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1070.040679] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1075.517351] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 1079.266352] LustreError: 35030:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1079.297896] LustreError: 35030:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1084.396292] LustreError: 35031:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1085.694909] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1085.955698] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1091.445525] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 1100.905647] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1107.346581] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1114.194930] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1114.530199] LustreError: 36850:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1114.766392] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1121.805678] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 1130.488288] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 1130.740479] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1147.737754] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 09:29:57 (1788442197) [ 1149.692231] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1151.876127] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1160.428542] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 09:30:09 (1788442209) [ 1170.554545] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 09:30:20 (1788442220) [ 1171.615454] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1181.267885] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1181.277978] Lustre: Skipped 2 previous similar messages [ 1183.511739] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 09:30:32 (1788442232) [ 1185.709291] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1185.713166] Lustre: Skipped 2 previous similar messages [ 1194.073697] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1194.083318] Lustre: Skipped 4 previous similar messages [ 1209.343672] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 09:30:59 (1788442259) [ 1212.312447] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1215.859582] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1225.331413] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 09:31:14 (1788442274) [ 1227.244511] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1227.249508] Lustre: Skipped 6 previous similar messages [ 1240.356992] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 09:31:30 (1788442290) [ 1258.470716] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1258.477551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1258.477610] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1258.493365] Lustre: Skipped 1 previous similar message [ 1258.525591] Lustre: Skipped 1 previous similar message [ 1261.081921] Lustre: server umount lustre-OST0000 complete [ 1268.712782] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1268.718568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1268.719340] Lustre: Skipped 1 previous similar message [ 1268.726140] Lustre: Skipped 1 previous similar message [ 1271.665981] Lustre: server umount lustre-MDT0000 complete [ 1273.824317] LustreError: 35031:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1273.837697] LustreError: 35031:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1276.629728] LustreError: 35011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442327 with bad export cookie 5089285967555440720 [ 1276.640227] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1276.657408] LustreError: 35011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1276.924600] Lustre: server umount lustre-MDT0001 complete [ 1288.855438] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 1292.505916] Key type lgssc unregistered [ 1292.825897] LNet: 42894:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1292.834394] LNetError: 42894:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1292.855072] LNet: Removed LNI 192.168.201.160@tcp [ 1294.042193] Key type .llcrypt unregistered [ 1294.048812] Key type ._llcrypt unregistered [ 1313.783880] Key type ._llcrypt registered [ 1313.786643] Key type .llcrypt registered [ 1313.906130] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_hostid [ 1331.690990] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 1332.915207] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1332.953653] alg: No test for adler32 (adler32-zlib) [ 1334.114480] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1334.441777] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [ 1336.208631] Key type lgssc registered [ 1337.236971] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1373.147317] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1373.179886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1374.503536] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1374.547390] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1374.648824] Lustre: lustre-MDT0000: new disk, initializing [ 1374.777896] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1374.813322] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1379.221194] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 1390.030955] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1390.166622] Lustre: 46051:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1390.217800] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1390.221902] Lustre: Skipped 1 previous similar message [ 1390.351294] Lustre: lustre-MDT0001: new disk, initializing [ 1390.513173] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1390.562349] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1390.573423] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1395.791355] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 1406.538547] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1412.808816] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1419.678499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1419.964451] Lustre: lustre-OST0000: new disk, initializing [ 1419.967821] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1419.980250] Lustre: 48020:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1420.091739] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1426.346067] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 1428.061754] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1428.068596] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1428.175876] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1436.438594] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1441.259372] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1441.474722] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1445.620545] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1445.815841] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1448.932884] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1448.934927] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1448.936987] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1448.944900] Lustre: Skipped 1 previous similar message [ 1453.752444] Lustre: server umount lustre-OST0000 complete [ 1459.170204] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1459.176228] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1459.189437] Lustre: Skipped 1 previous similar message [ 1459.202771] Lustre: Skipped 2 previous similar messages [ 1464.158835] Lustre: server umount lustre-MDT0000 complete [ 1468.089041] LustreError: 48019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442519 with bad export cookie 9718493581831045753 [ 1468.091801] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1468.108588] LustreError: 48019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1468.714685] Lustre: server umount lustre-MDT0001 complete [ 1477.802755] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 09:35:27 (1788442527) [ 1484.216484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1484.549063] LustreError: 50478:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1484.568657] LustreError: 50478:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1484.608274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1488.383312] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 1489.899653] LustreError: 50479:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1495.015285] LustreError: 50478:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1496.433856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1496.959290] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1501.461448] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 1509.270467] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1515.046523] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1521.624452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1522.047662] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1528.833666] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 1536.871779] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1545.059916] LustreError: 53007:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1545.997555] Lustre: Modifying parameter general.timeout=1 in log params [ 1547.027842] Lustre: Modifying parameter general.timeout=2 in log params [ 1549.166549] Lustre: Modifying parameter general.timeout=4 in log params [ 1549.171417] Lustre: Skipped 1 previous similar message [ 1553.761073] Lustre: Modifying parameter general.timeout=9 in log params [ 1553.768820] Lustre: Skipped 4 previous similar messages [ 1562.376357] Lustre: Modifying parameter general.timeout=18 in log params [ 1562.381066] Lustre: Skipped 8 previous similar messages [ 1578.800513] Lustre: Modifying parameter general.timeout=36 in log params [ 1578.803415] Lustre: Skipped 17 previous similar messages [ 1611.023786] Lustre: Modifying parameter general.timeout=72 in log params [ 1611.037540] Lustre: Skipped 35 previous similar messages [ 1677.624861] Lustre: Disabling parameter general.timeout= in log params [ 1677.635263] Lustre: Skipped 57 previous similar messages [ 1681.151799] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 09:38:50 (1788442730) [ 1684.203640] Lustre: 59949:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1686.262572] Lustre: 60047:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1688.264237] Lustre: 60159:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1694.146511] Lustre: 60470:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1694.168361] Lustre: 60470:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1720.755928] Lustre: 61832:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1720.777736] LustreError: 61832:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1725.325602] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 09:39:35 (1788442775) [ 1746.922281] Lustre: server umount lustre-MDT0000 complete [ 1747.949481] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1747.962192] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1747.986346] LustreError: 50475:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1747.995017] LustreError: Skipped 1 previous similar message [ 1748.039778] LustreError: 50475:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1753.068382] LustreError: 50474:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1753.100459] LustreError: 50474:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1753.139536] LustreError: 50460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442804 with bad export cookie 9718493581831046628 [ 1753.148049] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1753.160657] LustreError: 50460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1754.089108] Lustre: server umount lustre-MDT0001 complete [ 1768.831234] Lustre: server umount lustre-OST0000 complete [ 1801.590097] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1801.975787] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1802.414806] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1807.589990] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 1818.241204] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1818.396175] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1818.478651] Lustre: 64362:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1823.139581] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 1831.742495] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1838.440149] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1846.552231] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1846.768397] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1847.324594] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1847.333720] Lustre: Skipped 1 previous similar message [ 1854.612568] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 1863.047531] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1864.895662] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1864.901703] Lustre: Skipped 23 previous similar messages [ 1864.991383] Lustre: 66929:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1876.973151] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1876.979662] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1876.987797] LustreError: Skipped 1 previous similar message [ 1876.995050] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1877.000077] Lustre: Skipped 3 previous similar messages [ 1877.025906] Lustre: Skipped 1 previous similar message [ 1878.442656] Lustre: server umount lustre-OST0000 complete [ 1885.665772] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1885.676889] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1885.703070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1888.945841] Lustre: server umount lustre-MDT0000 complete [ 1892.326321] LustreError: 64371:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1892.370614] LustreError: 64371:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1893.788340] LustreError: 64355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442944 with bad export cookie 9718493581831049708 [ 1893.805891] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1894.319482] Lustre: server umount lustre-MDT0001 complete [ 1907.784822] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 1911.174262] Key type lgssc unregistered [ 1911.451603] LNet: 68140:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1911.462492] LNetError: 68140:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1911.477659] LNet: Removed LNI 192.168.201.160@tcp [ 1912.592606] Key type .llcrypt unregistered [ 1912.595390] Key type ._llcrypt unregistered [ 1942.689373] Key type ._llcrypt registered [ 1942.692238] Key type .llcrypt registered [ 1943.005554] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 1945.358751] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1945.389786] alg: No test for adler32 (adler32-zlib) [ 1946.737656] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1946.971739] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [ 1948.687267] Key type lgssc registered [ 1950.482110] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1962.729732] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1962.744473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1964.103569] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1970.572987] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 1973.217464] LustreError: 69558:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1973.241967] LustreError: 69558:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1978.338937] LustreError: 69559:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1981.578133] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1982.397909] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1988.395297] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 1997.224188] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2003.051838] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2010.274146] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2011.245465] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2018.285563] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 2027.291406] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2039.724569] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2049.508717] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2049.518515] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2049.524088] Lustre: Skipped 1 previous similar message [ 2049.540266] Lustre: Skipped 2 previous similar messages [ 2054.628279] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2054.647427] Lustre: Skipped 2 previous similar messages [ 2055.794324] Lustre: server umount lustre-MDT0000 complete [ 2059.747472] LustreError: 69558:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2059.775245] LustreError: 69558:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2060.396718] LustreError: 69539:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443111 with bad export cookie 13748375637160269306 [ 2060.399162] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2060.410173] LustreError: 69539:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2060.857317] Lustre: server umount lustre-MDT0001 complete [ 2075.712244] Lustre: server umount lustre-OST0000 complete [ 2108.156359] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2108.486336] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2108.521710] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2108.816181] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2114.654099] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 2125.576935] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2125.762105] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2125.791670] Lustre: 74433:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2126.433851] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2131.975988] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 2140.852114] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2146.564778] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2154.877984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2155.109561] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2155.650859] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2163.023058] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 2172.513709] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2175.461546] Lustre: 77010:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2178.102735] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2186.218178] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2186.225990] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2186.228789] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2186.238904] Lustre: Skipped 2 previous similar messages [ 2189.579829] Lustre: server umount lustre-OST0000 complete [ 2196.449169] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2196.464019] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2196.467689] Lustre: Skipped 1 previous similar message [ 2200.442707] Lustre: server umount lustre-MDT0000 complete [ 2201.572291] LustreError: 76924:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2201.604082] LustreError: 76924:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2204.425277] LustreError: 74426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443255 with bad export cookie 13748375637160270153 [ 2204.431091] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2204.434924] LustreError: 74426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2204.905448] Lustre: server umount lustre-MDT0001 complete [ 2217.037635] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 2219.908396] Key type lgssc unregistered [ 2220.185210] LNet: 78223:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2220.196913] LNetError: 78223:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2220.209234] LNet: Removed LNI 192.168.201.160@tcp [ 2220.991606] Key type .llcrypt unregistered [ 2220.993849] Key type ._llcrypt unregistered [ 2246.653217] Key type ._llcrypt registered [ 2246.657240] Key type .llcrypt registered [ 2246.779762] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 2248.069719] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2248.120259] alg: No test for adler32 (adler32-zlib) [ 2249.133590] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2249.388794] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [ 2251.051308] Key type lgssc registered [ 2252.210084] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2260.148847] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2260.172604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2261.909700] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2266.214237] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 2271.200768] LustreError: 79639:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2271.207304] LustreError: 79639:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2275.655454] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2276.307871] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2281.168917] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 2289.506574] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2296.369749] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2303.805409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2304.350303] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2311.055743] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 2318.866364] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2325.949634] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2327.098447] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2329.337408] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2329.344824] Lustre: Skipped 1 previous similar message [ 2331.772948] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2331.784218] Lustre: Skipped 1 previous similar message [ 2336.648718] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2336.653423] Lustre: Skipped 3 previous similar messages [ 2344.916695] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2344.938149] Lustre: Skipped 5 previous similar messages [ 2361.039161] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2361.043922] Lustre: Skipped 11 previous similar messages [ 2393.531344] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2393.536970] Lustre: Skipped 24 previous similar messages [ 2452.372518] Lustre: 87758:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2455.426293] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 09:51:45 (1788443505) [ 2457.946213] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2460.567434] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 09:51:49 (1788443509) [ 2465.248850] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2465.254261] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2465.261985] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2467.298044] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2467.300427] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2471.597865] Lustre: server umount lustre-OST0000 complete [ 2476.000312] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2476.010744] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2476.035587] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2476.040367] Lustre: Skipped 1 previous similar message [ 2482.113853] Lustre: server umount lustre-MDT0000 complete [ 2482.661514] LustreError: 80973:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2482.666776] LustreError: 80973:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2486.004333] LustreError: 79619:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443536 with bad export cookie 7213410630319338833 [ 2486.010067] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2486.026848] LustreError: 79619:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2486.496338] Lustre: server umount lustre-MDT0001 complete [ 2499.815954] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 2503.878764] Key type lgssc unregistered [ 2504.214721] LNet: 89010:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2504.229919] LNetError: 89010:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2504.253210] LNet: Removed LNI 192.168.201.160@tcp [ 2505.325104] Key type .llcrypt unregistered [ 2505.330859] Key type ._llcrypt unregistered [ 2510.405384] Key type ._llcrypt registered [ 2510.407674] Key type .llcrypt registered [ 2510.575372] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2515.737768] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules [ 2516.876968] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2516.906303] alg: No test for adler32 (adler32-zlib) [ 2518.018617] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2518.286131] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [ 2519.945573] Key type lgssc registered [ 2521.737864] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2523.795041] LustreError: 90252:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 2546.882314] LustreError: 90252:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2547.896956] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 2558.473308] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2558.486238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2560.115491] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2565.798789] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 2569.713503] LustreError: 91281:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2569.741134] LustreError: 91281:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2574.817554] LustreError: 91282:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2576.974437] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 09:53:46 (1788443626) [ 2579.946557] LustreError: 91281:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2582.900210] LustreError: 91313:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2582.923327] LustreError: 91313:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2582.945189] LustreError: 91313:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 22, retries 0, failed: rc = -5 [ 2583.494645] Lustre: server umount lustre-MDT0000 complete [ 2600.677158] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 2603.440602] Key type lgssc unregistered [ 2603.723948] LNet: 92812:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2603.740290] LNetError: 92812:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2603.759760] LNet: Removed LNI 192.168.201.160@tcp [ 2605.060928] Key type .llcrypt unregistered [ 2605.063908] Key type ._llcrypt unregistered [ 2625.280994] Key type ._llcrypt registered [ 2625.284882] Key type .llcrypt registered [ 2625.462908] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 2626.808523] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2626.891915] alg: No test for adler32 (adler32-zlib) [ 2628.009231] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2628.259604] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [ 2629.943284] Key type lgssc registered [ 2631.022130] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2638.953431] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2638.973611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2640.452728] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2645.321229] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 2650.593765] LustreError: 94032:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2650.614379] LustreError: 94032:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2655.725650] LustreError: 94033:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2656.659277] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2657.950226] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2664.051561] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 2673.059844] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2679.748747] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2688.341609] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2689.038121] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2695.432347] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 2703.461974] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2763.556749] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 09:56:52 (1788443812) [ 2765.285994] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2767.368827] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 09:56:57 (1788443817) [ 2771.949207] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2771.960476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2771.966839] Lustre: Skipped 2 previous similar messages [ 2771.971418] Lustre: Skipped 1 previous similar message [ 2777.062421] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2777.076083] Lustre: Skipped 3 previous similar messages [ 2778.294076] Lustre: server umount lustre-MDT0000 complete [ 2782.049378] LustreError: 94013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443832 with bad export cookie 8259586502707929394 [ 2782.056792] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2782.062174] LustreError: 94013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2782.180772] LustreError: 95883:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2782.181768] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2782.219281] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2782.341733] Lustre: server umount lustre-MDT0001 complete [ 2788.504971] Lustre: server umount lustre-OST0000 complete [ 2800.497766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2801.045882] LustreError: 98331:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2801.323898] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2806.457438] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 2806.768531] LustreError: 98332:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2814.947530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2815.580448] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2820.271754] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 2827.867803] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2834.128515] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2846.977829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2847.122909] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2847.131062] LustreError: 98321:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2847.150701] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2847.159411] LustreError: 100413:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2847.164730] LustreError: 100413:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2847.169968] LustreError: 100413:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2847.254863] Lustre: server umount lustre-OST0000 complete [ 2847.258984] LustreError: 100413:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2855.149478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2855.426122] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2855.433769] LustreError: 98320:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2855.459500] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2855.474402] LustreError: 100877:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2855.481864] LustreError: 100877:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2855.490574] LustreError: 100877:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2855.563631] Lustre: server umount lustre-OST0000 complete [ 2855.565772] LustreError: 100877:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2865.911372] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2866.106506] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2866.111879] Lustre: Found index 0 for lustre-OST0000, updating log [ 2866.116809] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2866.175272] Lustre: lustre-OST0000: new disk, initializing [ 2866.179101] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2866.184680] Lustre: 101411:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2866.560349] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2872.000523] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 2875.574389] Lustre: Failing over lustre-OST0000 [ 2875.702368] Lustre: server umount lustre-OST0000 complete [ 2880.997992] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2881.008929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2881.009737] Lustre: Skipped 1 previous similar message [ 2881.031559] Lustre: Skipped 1 previous similar message [ 2885.152489] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2885.160685] Lustre: Skipped 1 previous similar message [ 2889.301637] Lustre: server umount lustre-MDT0000 complete [ 2890.210647] LustreError: 98327:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2890.232781] LustreError: 98327:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2893.542482] LustreError: 98311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443944 with bad export cookie 8259586502707932138 [ 2893.548294] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2893.557182] LustreError: 98311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2893.916573] Lustre: server umount lustre-MDT0001 complete [ 2908.587420] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 09:59:17 (1788443957) [ 2928.716448] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 2942.229210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2942.634143] LustreError: 103758:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2942.753078] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2949.264556] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 2961.035538] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2966.754550] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 2970.717276] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2978.753458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2979.649963] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2979.660757] Lustre: Skipped 1 previous similar message [ 2979.677988] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2986.410582] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 2986.788738] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2986.816943] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2986.837991] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2987.815032] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:08 [ 2987.852396] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2987.852820] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2987.887569] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2987.891667] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2997.964835] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2998.050589] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2998.144886] Lustre: lustre-OST0001: new disk, initializing [ 2998.156282] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2998.164276] Lustre: 106078:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3007.068954] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 3007.760537] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3007.784866] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3007.888713] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3017.167598] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3027.306700] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3033.568588] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 3033.570147] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3033.573503] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3033.602681] Lustre: Skipped 2 previous similar messages [ 3039.769707] Lustre: server umount lustre-MDT0001 complete [ 3042.018937] LustreError: 103755:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3042.082729] LustreError: 103755:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3051.436622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3051.565996] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3051.576190] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3051.654642] Lustre: 103748:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3052.699239] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3052.713827] Lustre: Skipped 1 previous similar message [ 3053.091926] LustreError: 107627:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3053.109283] LustreError: 107627:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3053.123389] LustreError: 103750:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3053.258155] LustreError: 107637:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.160@tcp: cfg command failed: rc = -17 [ 3053.279879] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3053.279879] [ 3057.141313] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 3058.161415] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3067.710860] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 10:01:57 (1788444117) [ 3089.848323] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 3110.224971] Lustre: 109710:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3137.340556] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3141.991770] Lustre: 110845:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3236.321361] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3236.324895] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3236.341598] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3236.350158] Lustre: Skipped 6 previous similar messages [ 3241.988762] Lustre: server umount lustre-MDT0000 complete [ 3242.464511] LustreError: 103755:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3242.477704] LustreError: 103755:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 3247.077472] LustreError: 106812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444297 with bad export cookie 8259586502707933069 [ 3247.082603] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3247.090503] LustreError: 106812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 3247.587953] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3247.610095] Lustre: Skipped 3 previous similar messages [ 3247.999047] Lustre: server umount lustre-MDT0001 complete [ 3254.019330] Lustre: server umount lustre-OST0000 complete [ 3260.255506] Lustre: server umount lustre-OST0001 complete [ 3267.655134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3278.677634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3287.818059] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3298.021907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3321.589262] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 3333.586395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3333.602054] Lustre: lustre-MDT0000: reset Object Index mappings [ 3334.262045] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3338.708347] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 3347.846263] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3347.900960] Lustre: lustre-MDT0001: reset Object Index mappings [ 3354.503529] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 3358.107581] Lustre: 115944:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3365.714174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3373.243774] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 3373.690518] LustreError: 116299:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3373.700369] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:199 to 0x280000400:225) [ 3373.706290] LustreError: 116299:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 3380.238758] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:381 to 0x280000401:418) [ 3385.252692] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3391.520359] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 3391.543138] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:157 to 0x2c0000401:193) [ 3394.315665] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 3403.301469] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3408.268845] Lustre: 117822:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3464.115843] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 10:08:33 (1788444513) [ 3491.814588] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3491.828236] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3491.861683] Lustre: Skipped 1 previous similar message [ 3491.872277] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3491.878349] Lustre: Skipped 4 previous similar messages [ 3497.980842] Lustre: server umount lustre-MDT0000 complete [ 3502.596523] LustreError: 114778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444553 with bad export cookie 8259586502707980144 [ 3502.619322] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3502.622527] LustreError: 114778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3509.350154] Lustre: server umount lustre-MDT0001 complete [ 3520.301751] Lustre: server umount lustre-OST0000 complete [ 3538.399138] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3538.696490] Lustre: server umount lustre-OST0001 complete [ 3546.139720] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_hostid [ 3561.150733] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 3601.771972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3602.034149] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3602.062502] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3602.136405] Lustre: lustre-MDT0000: new disk, initializing [ 3602.199438] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3602.207742] Lustre: Skipped 3 previous similar messages [ 3602.229410] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3606.784997] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 3614.939072] Lustre: server umount lustre-MDT0000 complete [ 3626.294200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3626.799812] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3626.841940] Lustre: 122123:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3626.858482] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3633.486295] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 3639.330321] Lustre: server umount lustre-MDT0000 complete [ 3649.297545] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 10:11:38 (1788444698) [ 3651.612246] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3654.308604] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 10:11:43 (1788444703) [ 3681.437532] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 3694.984271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3701.029133] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 3711.380928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3711.596097] Lustre: 124197:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3711.717705] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3711.732220] Lustre: Skipped 1 previous similar message [ 3711.858443] Lustre: lustre-MDT0001: new disk, initializing [ 3711.963440] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3711.970522] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3717.036761] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 3722.163986] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3729.690214] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3730.153392] Lustre: lustre-OST0000: new disk, initializing [ 3730.162268] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3730.185219] Lustre: 125774:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3731.848022] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3731.865493] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3732.062620] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3737.981566] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 3750.364634] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3750.542950] Lustre: lustre-OST0001: new disk, initializing [ 3750.551362] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3750.569996] Lustre: 126648:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3751.817409] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3751.834071] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3751.992971] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3757.599568] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 3769.384141] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3773.912129] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4169.479577] Lustre: 133063:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4169.483637] Lustre: 133063:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4169.487167] Lustre: 133063:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4169.493208] Lustre: 133063:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4169.498792] Lustre: 133063:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4169.503170] Lustre: 133063:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4459.776635] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 10:25:09 (1788445509) [ 4475.360790] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4475.364416] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4475.399193] Lustre: Skipped 5 previous similar messages [ 4475.403529] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4475.406244] Lustre: Skipped 8 previous similar messages [ 4480.647681] Lustre: server umount lustre-MDT0000 complete [ 4484.070669] LustreError: 125769:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4484.091599] LustreError: 125769:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 16 previous similar messages [ 4484.593608] LustreError: 126646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445535 with bad export cookie 8259586502708049199 [ 4484.599276] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4484.604063] LustreError: 126646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 4489.187271] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4489.209148] Lustre: Skipped 4 previous similar messages [ 4489.211667] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4489.230674] Lustre: Skipped 5 previous similar messages [ 4491.358665] Lustre: server umount lustre-MDT0001 complete [ 4501.724476] Lustre: server umount lustre-OST0000 complete [ 4512.329701] Lustre: server umount lustre-OST0001 complete [ 4521.602518] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_hostid [ 4537.571451] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 4582.734059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4582.922263] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4582.943477] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4583.011435] Lustre: lustre-MDT0000: new disk, initializing [ 4583.082920] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4583.087439] Lustre: Skipped 5 previous similar messages [ 4583.103446] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4587.685793] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 4599.062662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4599.208711] Lustre: 136136:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4599.268973] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4599.274567] Lustre: Skipped 1 previous similar message [ 4599.382325] Lustre: lustre-MDT0001: new disk, initializing [ 4599.565352] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4599.582597] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4604.397333] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 4614.800303] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4620.364622] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4627.647672] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4627.967786] Lustre: lustre-OST0000: new disk, initializing [ 4627.971463] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4627.980682] Lustre: 138103:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4629.436374] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4629.445787] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4629.530788] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4634.888203] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 4644.827323] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4655.829152] Lustre: lustre-MDD0000: changelog on [ 4659.040386] Lustre: lustre-MDD0001: changelog on [ 6586.673660] Lustre: lustre-MDD0001: changelog off [ 6589.110241] Lustre: 138431:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6589.834956] Lustre: lustre-MDD0000: changelog off [ 6595.555446] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6595.556250] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6595.558907] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6595.564970] LustreError: Skipped 1 previous similar message [ 6595.578432] Lustre: Skipped 1 previous similar message [ 6600.676434] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6600.687589] Lustre: Skipped 1 previous similar message [ 6605.802689] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6605.813592] Lustre: Skipped 1 previous similar message [ 6608.884920] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6609.182819] Lustre: server umount lustre-OST0000 complete [ 6616.032427] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6616.044864] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6616.048220] Lustre: Skipped 1 previous similar message [ 6618.978068] Lustre: server umount lustre-MDT0000 complete [ 6621.152858] LustreError: 136147:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6621.173688] LustreError: 136147:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6622.523884] LustreError: 136128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447673 with bad export cookie 8259586502708831211 [ 6622.525214] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6622.532769] LustreError: 136128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6622.854920] Lustre: server umount lustre-MDT0001 complete [ 6635.621691] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 6638.324548] Key type lgssc unregistered [ 6638.684153] LNet: 147279:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6638.693370] LNetError: 147279:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6638.713768] LNet: Removed LNI 192.168.201.160@tcp [ 6639.487165] Key type .llcrypt unregistered [ 6639.489408] Key type ._llcrypt unregistered [ 6658.158763] Key type ._llcrypt registered [ 6658.160576] Key type .llcrypt registered [ 6658.239661] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_hostid [ 6671.989228] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 6672.993885] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6673.113064] alg: No test for adler32 (adler32-zlib) [ 6674.099554] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 6674.276374] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [ 6675.921934] Key type lgssc registered [ 6677.008227] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6710.431212] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6710.461483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6711.732378] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6711.757652] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6711.803299] Lustre: lustre-MDT0000: new disk, initializing [ 6711.858732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6711.866850] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6715.595668] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 6725.988279] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6726.141604] Lustre: 150479:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6726.177180] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6726.182128] Lustre: Skipped 1 previous similar message [ 6726.253570] Lustre: lustre-MDT0001: new disk, initializing [ 6726.358035] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6726.405964] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6726.429168] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6730.657401] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 6739.880824] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6745.660271] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6752.804156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6753.069525] Lustre: lustre-OST0000: new disk, initializing [ 6753.075530] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6753.083889] Lustre: 152447:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6753.177063] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6759.919516] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 6761.067165] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6761.076714] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6761.222877] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6771.721571] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6778.159438] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_module obdecho/obdecho [ 6780.379849] LustreError: 153450:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6780.398538] LustreError: 153450:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6782.461869] LustreError: 153547:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6797.283047] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6797.298807] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6797.300492] Lustre: Skipped 1 previous similar message [ 6800.792030] Lustre: server umount lustre-MDT0000 complete [ 6804.311169] LustreError: 150473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447855 with bad export cookie 1592730831746218626 [ 6804.314167] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6804.321174] LustreError: 150473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6804.557723] Lustre: server umount lustre-MDT0001 complete [ 6818.817313] Lustre: server umount lustre-OST0000 complete [ 6842.271834] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_hostid [ 6854.009456] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 6896.079984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6896.327231] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6896.354392] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6896.433503] Lustre: lustre-MDT0000: new disk, initializing [ 6896.473200] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6896.482834] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6901.200505] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 6913.074706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6913.304722] Lustre: 157263:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6913.411799] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6913.419633] Lustre: Skipped 1 previous similar message [ 6913.576627] Lustre: lustre-MDT0001: new disk, initializing [ 6913.759455] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6913.793692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6913.799730] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6918.409402] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 6928.285818] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6933.695370] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6939.868410] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6940.058705] Lustre: lustre-OST0000: new disk, initializing [ 6940.062595] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6940.068935] Lustre: 159229:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6940.150135] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6946.430225] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 6947.969501] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6947.987989] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6948.178087] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6956.249285] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6963.128412] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6963.660930] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6968.827418] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6968.988457] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6973.923812] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6973.930189] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6973.937235] Lustre: Skipped 1 previous similar message [ 6973.939247] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6973.943255] LustreError: Skipped 1 previous similar message [ 6973.963626] Lustre: Skipped 3 previous similar messages [ 6976.872859] Lustre: server umount lustre-OST0000 complete [ 6984.165554] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6984.167483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6984.174845] Lustre: Skipped 2 previous similar messages [ 6984.184365] Lustre: Skipped 1 previous similar message [ 6987.390207] Lustre: server umount lustre-MDT0000 complete [ 6989.284722] LustreError: 157274:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6989.337938] LustreError: 157274:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6991.465587] LustreError: 157256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448042 with bad export cookie 1592730831746219585 [ 6991.477131] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6991.497032] LustreError: 157256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6992.181907] Lustre: server umount lustre-MDT0001 complete [ 6996.388498] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 11:07:26 (1788448046) [ 6998.405095] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7000.612127] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 11:07:30 (1788448050) [ 7010.514701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7011.024686] LustreError: 161644:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7011.135573] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7015.694580] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 7016.417284] LustreError: 161645:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7020.523769] LustreError: 161644:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7024.924858] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7030.398470] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 7039.265374] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7045.548882] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7053.178760] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7053.578796] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7053.588785] Lustre: Skipped 1 previous similar message [ 7060.246344] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 7067.523835] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7078.656368] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.60@tcp (stopping) [ 7079.239486] Lustre: server umount lustre-MDT0000 complete [ 7081.445551] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7081.460179] Lustre: Skipped 1 previous similar message [ 7081.463035] LustreError: 161640:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7081.495172] LustreError: 161640:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7082.540276] LustreError: 161625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448133 with bad export cookie 1592730831746220467 [ 7082.542101] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7082.549548] LustreError: 161625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7083.776994] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.60@tcp (stopping) [ 7086.569900] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7088.937062] Lustre: server umount lustre-MDT0001 complete [ 7095.128337] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7109.141548] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7109.499948] LustreError: 165488:0:(llog.c:1655:llog_backup()) MGC192.168.201.160@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7109.516810] Lustre: 165488:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.160@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7112.164924] LustreError: 165493:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7112.190491] LustreError: 165493:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7130.849172] LustreError: 165493:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7130.868391] LustreError: 165493:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 7165.667290] LustreError: 165494:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7165.697554] LustreError: 165494:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 27 previous similar messages [ 7170.528121] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7170.536769] Lustre: 165488:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.160@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7211.522550] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7216.608590] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 7216.618741] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7226.250023] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7227.423130] LustreError: 166230:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 7227.427084] LustreError: 166230:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@00000000ee184936 x1875323361816064/t0(0) o250->MGC192.168.201.160@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1788448278 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7227.451882] LustreError: 166230:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 7227.872794] LustreError: 148702:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000075bf914c x1875323361821056/t0(0) o250->MGC192.168.201.160@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7232.515879] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 7233.520682] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7241.039627] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7242.747749] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7248.784181] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7250.384788] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7272.310696] Lustre: server umount lustre-MDT0000 complete [ 7274.465017] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7274.477571] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7274.480971] LustreError: 163479:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7274.480981] LustreError: 163479:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 45 previous similar messages [ 7274.520153] Lustre: Skipped 2 previous similar messages [ 7276.543383] LustreError: 161625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448327 with bad export cookie 1592730831746226921 [ 7276.544447] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7276.549231] LustreError: 161625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7276.809767] Lustre: server umount lustre-MDT0001 complete [ 7290.961918] Lustre: server umount lustre-OST0000 complete [ 7312.805585] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_hostid [ 7327.452057] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 7371.923753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7372.185456] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7372.236313] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7372.372682] Lustre: lustre-MDT0000: new disk, initializing [ 7372.484506] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7372.490910] Lustre: Skipped 1 previous similar message [ 7372.542684] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7377.359356] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 7387.384637] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7387.567988] Lustre: 170621:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7388.062967] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7392.771771] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 7403.250337] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7409.716388] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7416.493540] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7416.830555] Lustre: lustre-OST0000: new disk, initializing [ 7416.833672] Lustre: Skipped 1 previous similar message [ 7416.837317] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7416.841496] Lustre: Skipped 2 previous similar messages [ 7416.849095] Lustre: 172589:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7422.951798] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 7425.127154] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7425.134059] Lustre: Skipped 1 previous similar message [ 7425.144535] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7425.303579] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7433.173193] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7437.985829] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7438.163911] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7442.933317] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7443.127561] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7445.992838] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7445.999503] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7446.013545] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7446.032761] Lustre: Skipped 3 previous similar messages [ 7459.295814] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7459.555503] Lustre: server umount lustre-OST0000 complete [ 7464.928992] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7464.941844] LustreError: Skipped 1 previous similar message [ 7464.953290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7464.964202] Lustre: Skipped 4 previous similar messages [ 7470.087135] Lustre: server umount lustre-MDT0000 complete [ 7471.585058] LustreError: 170632:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7471.614618] LustreError: 170632:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7473.564206] LustreError: 170614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448524 with bad export cookie 1592730831746243756 [ 7473.566532] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7473.575365] LustreError: 170614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7473.964366] Lustre: server umount lustre-MDT0001 complete [ 7481.963672] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 11:15:31 (1788448531) [ 7488.173220] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7492.519793] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 7500.488305] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7504.622333] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 7511.626664] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7517.553816] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7525.642926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7533.434177] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 7540.365984] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7552.979510] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 11:16:42 (1788448602) [ 7558.625669] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7558.633354] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7558.649217] Lustre: Skipped 3 previous similar messages [ 7558.656688] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7558.664602] Lustre: Skipped 2 previous similar messages [ 7562.907981] Lustre: server umount lustre-OST0000 complete [ 7567.328598] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7567.335493] LustreError: Skipped 1 previous similar message [ 7572.878956] Lustre: server umount lustre-MDT0000 complete [ 7576.342597] LustreError: 174982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448627 with bad export cookie 1592730831746244638 [ 7576.348579] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7576.351994] LustreError: 174982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7576.762198] Lustre: server umount lustre-MDT0001 complete [ 7588.454531] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 7591.399340] Key type lgssc unregistered [ 7591.670930] LNet: 178733:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7591.679102] LNetError: 178733:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7591.698449] LNet: Removed LNI 192.168.201.160@tcp [ 7592.795215] Key type .llcrypt unregistered [ 7592.796952] Key type ._llcrypt unregistered [ 7613.833839] Key type ._llcrypt registered [ 7613.835718] Key type .llcrypt registered [ 7613.967136] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 7614.793502] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7614.846235] alg: No test for adler32 (adler32-zlib) [ 7616.018112] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 7616.348916] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [ 7618.018062] Key type lgssc registered [ 7619.770612] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7628.612249] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7628.624945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7644.127721] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7649.247541] LustreError: 179983:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.160@tcp: failed processing log, type 4: rc = -110 [ 7654.367488] LustreError: 179983:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7654.389910] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7654.395759] LustreError: MGC192.168.201.160@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7654.409080] LustreError: 179983:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7654.429690] LustreError: 179983:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 7654.457825] LustreError: 179983:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 7654.535092] Lustre: server umount lustre-OST0000 complete [ 7654.536891] LustreError: 179983:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7663.473299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7664.854459] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7669.290365] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 7674.337474] LustreError: 180482:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7674.358966] LustreError: 180482:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7677.422488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7677.827847] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7682.610167] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 7690.214471] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7696.403902] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7703.234168] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7703.436852] LustreError: 182302:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7703.466515] Lustre: 182302:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.201.160@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7703.628162] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7709.536113] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 7717.692232] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7720.417488] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7720.437913] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7722.539128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7722.556543] Lustre: Skipped 4 previous similar messages [ 7726.400608] Lustre: server umount lustre-MDT0000 complete [ 7727.590226] LustreError: 180483:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7727.609900] LustreError: 180483:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7729.913767] LustreError: 180462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448780 with bad export cookie 16403326489908033328 [ 7729.929996] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7730.783915] LustreError: 181212:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000005a5599df x1875324349477248/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7730.975804] Lustre: server umount lustre-MDT0001 complete [ 7744.637761] Lustre: server umount lustre-OST0000 complete [ 7754.075508] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_hostid [ 7764.904496] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 7800.413296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7800.636204] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7800.660934] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7800.738861] Lustre: lustre-MDT0000: new disk, initializing [ 7800.798184] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7800.822110] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7805.488977] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 7816.650139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7816.743407] Lustre: 185821:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7816.796122] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7816.799749] Lustre: Skipped 1 previous similar message [ 7816.864677] Lustre: lustre-MDT0001: new disk, initializing [ 7816.959743] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7816.991738] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7816.995267] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7821.521415] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 7832.058502] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7837.948576] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7846.142112] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7846.430288] Lustre: lustre-OST0000: new disk, initializing [ 7846.434650] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7846.443961] Lustre: 187794:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7846.522827] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7853.655693] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 7856.709948] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7856.716075] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7856.792444] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7863.608509] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7868.458624] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7868.658387] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7872.792380] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7872.998441] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7877.600423] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7877.603326] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7877.615767] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7877.631202] Lustre: Skipped 3 previous similar messages [ 7877.664687] Lustre: Skipped 1 previous similar message [ 7882.735324] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7882.754083] Lustre: Skipped 1 previous similar message [ 7887.843876] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7887.850379] Lustre: Skipped 1 previous similar message [ 7889.375133] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7889.572240] Lustre: server umount lustre-OST0000 complete [ 7893.986925] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7893.993815] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7898.081514] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7898.083218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7898.104561] Lustre: Skipped 2 previous similar messages [ 7899.584623] Lustre: server umount lustre-MDT0000 complete [ 7903.201337] LustreError: 185835:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7903.223458] LustreError: 185835:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7903.262215] LustreError: 186619:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448954 with bad export cookie 16403326489908034049 [ 7903.281922] LustreError: 186619:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 7903.282455] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7903.665649] Lustre: server umount lustre-MDT0001 complete [ 7912.821361] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 11:22:42 (1788448962) [ 7932.712193] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 7935.804859] Key type lgssc unregistered [ 7936.101482] LNet: 190742:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7936.113329] LNetError: 190742:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7936.125963] LNet: Removed LNI 192.168.201.160@tcp [ 7936.915312] Key type .llcrypt unregistered [ 7936.917859] Key type ._llcrypt unregistered [ 7955.872593] Key type ._llcrypt registered [ 7955.874634] Key type .llcrypt registered [ 7956.004397] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 7957.183491] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7957.367147] alg: No test for adler32 (adler32-zlib) [ 7958.520319] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 7958.797290] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [ 7960.471419] Key type lgssc registered [ 7961.962798] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7971.754848] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7971.801326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7973.205354] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7977.314817] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 7983.587824] LustreError: 191962:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7983.603664] LustreError: 191962:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7986.088673] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7986.388608] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7990.042572] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 7998.939570] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8005.602863] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8013.297446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8013.561622] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8013.564780] LustreError: 193826:0:(llog.c:1696:llog_backup()) MGC192.168.201.160@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8013.576651] Lustre: 193826:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.201.160@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8013.688850] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8019.139459] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 8026.774514] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8029.154986] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8029.166495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8029.178387] Lustre: Skipped 2 previous similar messages [ 8029.186216] Lustre: Skipped 1 previous similar message [ 8031.275115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8031.284767] Lustre: Skipped 3 previous similar messages [ 8034.958205] Lustre: server umount lustre-MDT0000 complete [ 8036.328040] LustreError: 191959:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8036.340991] LustreError: 191959:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8039.276659] LustreError: 191943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449090 with bad export cookie 10271978865891421592 [ 8039.306830] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8039.478366] LustreError: 192693:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000e5c369c3 x1875324708089088/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8040.143313] Lustre: server umount lustre-MDT0001 complete [ 8054.416433] Lustre: server umount lustre-OST0000 complete [ 8065.826248] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_hostid [ 8076.842829] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 8114.504303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8114.814788] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8114.847747] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8115.016141] Lustre: lustre-MDT0000: new disk, initializing [ 8115.138394] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8115.162604] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8120.486827] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 8129.981947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8130.126234] Lustre: 197350:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8130.172665] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8130.178932] Lustre: Skipped 1 previous similar message [ 8130.186212] LustreError: 198133:0:(llog.c:1696:llog_backup()) MGC192.168.201.160@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8130.201737] LustreError: 198133:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8130.215737] Lustre: 198133:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.160@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8130.252332] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8130.258202] Lustre: Skipped 1 previous similar message [ 8130.340454] Lustre: lustre-MDT0001: new disk, initializing [ 8130.475579] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8130.515906] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8130.534853] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8134.557240] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 8143.219398] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8148.431157] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8154.671493] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8154.989958] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8154.991636] Lustre: Skipped 1 previous similar message [ 8154.993341] LustreError: 199295:0:(llog.c:1696:llog_backup()) MGC192.168.201.160@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8155.000048] LustreError: 199295:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8155.005697] Lustre: 199295:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.160@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8155.011865] Lustre: 199295:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8155.039598] Lustre: lustre-OST0000: new disk, initializing [ 8155.042821] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8155.047688] Lustre: 199320:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8155.122656] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8160.353125] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8160.373462] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8160.494018] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8161.521561] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 8172.357188] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8178.500934] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8178.844605] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8184.602680] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8184.870818] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8191.456656] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8191.460901] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8191.480732] Lustre: Skipped 1 previous similar message [ 8196.585488] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8196.603094] Lustre: Skipped 2 previous similar messages [ 8201.187154] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8201.307322] Lustre: server umount lustre-OST0000 complete [ 8206.815968] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8206.828660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8206.838335] Lustre: Skipped 1 previous similar message [ 8211.930182] Lustre: server umount lustre-MDT0000 complete [ 8211.946779] LustreError: 197362:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8211.971759] LustreError: 197362:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8215.930620] LustreError: 198146:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449266 with bad export cookie 10271978865891422306 [ 8215.938622] LustreError: 198146:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8215.939140] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8216.289694] Lustre: server umount lustre-MDT0001 complete [ 8226.740678] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 11:27:56 (1788449276) [ 8244.927801] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 8254.954114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8255.357435] LustreError: 202450:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8255.390180] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8259.813258] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 8260.578757] LustreError: 202451:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8267.573900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8272.111568] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 8275.619322] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8282.718657] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8289.344664] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 8299.825376] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8300.077654] Lustre: lustre-OST0001: new disk, initializing [ 8300.080532] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8300.099268] Lustre: 204756:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8300.271581] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8300.281332] Lustre: Skipped 2 previous similar messages [ 8307.364216] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8307.380966] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8307.517456] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8307.805732] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 8318.874156] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8327.332559] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8329.382301] Lustre: DEBUG MARKER: ADD OST3 [ 8334.806485] loop: module loaded [ 8334.871824] loop0: detected capacity change from 0 to 204800000 [ 8335.114831] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8335.127126] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8335.136892] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8342.606347] Lustre: DEBUG MARKER: STOP OST3 [ 8342.730944] loop0: detected capacity change from 0 to 204800000 [ 8349.687261] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8349.920113] Lustre: lustre-OST0003: new disk, initializing [ 8349.930169] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8349.950236] Lustre: 206795:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8351.041113] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8351.241037] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8351.250942] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8351.286707] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8353.064661] LustreError: 206891:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8353.757544] Lustre: Failing over lustre-OST0003 [ 8353.946926] Lustre: server umount lustre-OST0003 complete [ 8356.327180] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8356.338749] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8356.354607] LustreError: 203936:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8356.354618] LustreError: 203936:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8356.395892] Lustre: Skipped 2 previous similar messages [ 8358.045037] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 8366.359328] Lustre: DEBUG MARKER: START OST3 again [ 8374.011588] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8374.226464] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8374.229645] Lustre: Skipped 1 previous similar message [ 8374.246518] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8375.525539] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8375.764031] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8375.766642] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8380.292719] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 8400.732910] Lustre: Failing over lustre-OST0003 [ 8400.947650] Lustre: server umount lustre-OST0003 complete [ 8401.387170] LustreError: lustre-OST0003-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8401.388358] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8401.394306] LustreError: Skipped 1 previous similar message [ 8401.398417] LustreError: 203936:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8401.406763] Lustre: Skipped 1 previous similar message [ 8401.441316] LustreError: 203936:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [ 8420.327802] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8420.340772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8420.343450] Lustre: Skipped 2 previous similar messages [ 8420.350826] Lustre: Skipped 1 previous similar message [ 8424.021087] Lustre: server umount lustre-MDT0000 complete [ 8427.194951] LustreError: 202432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449478 with bad export cookie 10271978865891423195 [ 8427.200971] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8427.209904] LustreError: 202432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8433.650829] Lustre: server umount lustre-MDT0001 complete [ 8437.765612] Lustre: server umount lustre-OST0000 complete [ 8441.638804] Lustre: server umount lustre-OST0001 complete [ 8448.480188] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_hostid [ 8456.871372] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 8498.203773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8498.481254] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8498.533815] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8498.634422] Lustre: lustre-MDT0000: new disk, initializing [ 8498.792473] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8503.221132] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 8514.765041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8514.889439] Lustre: 212212:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8514.927792] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8514.931424] Lustre: Skipped 1 previous similar message [ 8514.994497] Lustre: lustre-MDT0001: new disk, initializing [ 8515.125054] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8515.127751] Lustre: Skipped 1 previous similar message [ 8515.178452] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8515.194480] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8520.398983] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 8533.341328] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8539.876748] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8547.979527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8548.299286] Lustre: lustre-OST0000: new disk, initializing [ 8548.301690] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8548.305727] Lustre: 214182:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8549.841531] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8549.854463] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8549.978916] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8555.234266] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 8564.628794] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8569.439261] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8569.609523] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8573.354856] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8573.520769] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8575.460873] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8575.461250] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8575.464884] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8575.464893] Lustre: Skipped 6 previous similar messages [ 8575.469860] LustreError: Skipped 1 previous similar message [ 8575.493266] Lustre: Skipped 4 previous similar messages [ 8589.279187] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8589.553461] Lustre: server umount lustre-OST0000 complete [ 8600.407852] Lustre: server umount lustre-MDT0000 complete [ 8601.058052] LustreError: 213095:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8601.082112] LustreError: 213095:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 17 previous similar messages [ 8604.023292] LustreError: 212206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449654 with bad export cookie 10271978865891425491 [ 8604.044061] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8604.451561] Lustre: server umount lustre-MDT0001 complete [ 8608.411983] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 11:34:18 (1788449658) [ 8625.137126] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_hostid [ 8637.390687] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 8685.668387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8685.965136] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8685.990176] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8686.045531] Lustre: lustre-MDT0000: new disk, initializing [ 8686.150633] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8692.229860] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 8704.444097] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8704.537164] Lustre: 218799:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8704.734586] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8710.731530] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 8721.877754] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8728.305829] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8736.697146] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8737.057240] Lustre: 220769:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8738.328630] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8738.504706] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8745.410543] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 8758.622979] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8765.186730] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8765.415371] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8770.640246] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8770.974660] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8774.112992] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8774.122564] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8774.137644] Lustre: Skipped 2 previous similar messages [ 8774.142835] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8774.148505] Lustre: Skipped 7 previous similar messages [ 8779.753218] LustreError: 221390:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8779.762769] LustreError: 221390:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8779.858884] Lustre: server umount lustre-OST0000 complete [ 8786.912787] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8791.350894] Lustre: server umount lustre-MDT0000 complete [ 8795.597835] LustreError: 218791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449846 with bad export cookie 10271978865891426373 [ 8795.601190] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8795.607479] LustreError: 218791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 8796.008905] Lustre: server umount lustre-MDT0001 complete [ 8804.785414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8805.431827] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8805.440587] Lustre: Skipped 4 previous similar messages [ 8810.279702] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 8818.538722] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8823.495792] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 8831.550605] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8838.618343] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8846.796783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8854.994717] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 8863.600840] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8873.954626] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8873.965346] Lustre: Skipped 3 previous similar messages [ 8873.965682] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8873.968651] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8873.977768] Lustre: Skipped 5 previous similar messages [ 8879.969633] Lustre: server umount lustre-OST0000 complete [ 8889.743082] Lustre: server umount lustre-MDT0000 complete [ 8893.648538] LustreError: 222967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449944 with bad export cookie 10271978865891427255 [ 8893.655680] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8893.657177] LustreError: 222967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8894.211034] Lustre: server umount lustre-MDT0001 complete [ 9002.739791] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 9005.397852] Key type lgssc unregistered [ 9005.661373] LNet: 226654:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9005.667994] LNetError: 226654:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9006.697461] LNet: Removed LNI 192.168.201.160@tcp [ 9007.750209] Key type .llcrypt unregistered [ 9007.752590] Key type ._llcrypt unregistered [ 9034.974123] Key type ._llcrypt registered [ 9034.978763] Key type .llcrypt registered [ 9035.107961] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_hostid [ 9052.994918] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 9054.507942] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9054.533456] alg: No test for adler32 (adler32-zlib) [ 9055.609660] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 9055.875054] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [ 9057.575601] Key type lgssc registered [ 9058.670779] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9094.423955] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9094.452233] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9095.697321] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9095.719886] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9095.810816] Lustre: lustre-MDT0000: new disk, initializing [ 9095.891764] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9095.908648] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9100.442381] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 9112.008210] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9112.191241] Lustre: 230009:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9112.242643] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9112.246371] Lustre: Skipped 1 previous similar message [ 9112.364471] Lustre: lustre-MDT0001: new disk, initializing [ 9112.494832] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9112.540387] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9112.561224] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9117.006295] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 9126.113173] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9132.226318] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9139.103400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9139.403783] Lustre: lustre-OST0000: new disk, initializing [ 9139.409900] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9139.418932] Lustre: 231977:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9139.504994] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9144.980115] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 9149.000306] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9149.005423] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9149.103520] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9153.805914] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9158.443741] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9158.586279] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9162.490216] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9162.691215] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9164.768714] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9164.773787] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9164.786157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9164.813897] Lustre: Skipped 1 previous similar message [ 9169.892086] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9169.915186] Lustre: Skipped 2 previous similar messages [ 9170.542405] Lustre: server umount lustre-OST0000 complete [ 9173.994891] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9174.006466] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9174.027663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9175.010059] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9180.129836] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9180.140480] Lustre: Skipped 3 previous similar messages [ 9185.250098] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9185.259600] Lustre: Skipped 1 previous similar message [ 9188.320127] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9188.585470] Lustre: server umount lustre-MDT0000 complete [ 9191.993895] LustreError: 230804:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450242 with bad export cookie 8636047081621792359 [ 9191.998458] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9192.011307] LustreError: 230804:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9192.497142] Lustre: server umount lustre-MDT0001 complete [ 9213.271937] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 9222.988930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9223.443068] LustreError: 234912:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9223.468324] LustreError: 234912:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9223.510319] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9228.103785] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 9228.774422] LustreError: 234913:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9233.895262] LustreError: 234912:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9236.822168] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9237.204396] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9242.257752] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 9245.625897] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9252.995167] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9253.375835] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9259.041287] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 9267.716399] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9267.867743] Lustre: lustre-OST0001: new disk, initializing [ 9267.870742] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9267.875176] Lustre: 237218:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9274.595604] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 9277.067798] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9277.078961] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9277.199672] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9286.506070] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9294.922245] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9308.673894] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 11:45:58 (1788450358) [ 9313.255182] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9313.257781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9313.268178] Lustre: Skipped 2 previous similar messages [ 9313.276010] Lustre: Skipped 3 previous similar messages [ 9317.169439] Lustre: server umount lustre-MDT0000 complete [ 9318.375558] LustreError: 238069:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9318.385602] LustreError: 238069:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9320.856530] LustreError: 234893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450371 with bad export cookie 8636047081621793248 [ 9320.873052] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9320.874147] LustreError: 234893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9321.253513] Lustre: server umount lustre-MDT0001 complete [ 9335.504771] Lustre: server umount lustre-OST0000 complete [ 9339.873667] Lustre: 228234:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788450374/real 1788450374] req@ffff8d4ccd4e6d80 x1875325858460032/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788450390 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9339.896985] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9339.906643] Lustre: Skipped 1 previous similar message [ 9340.326952] Lustre: server umount lustre-OST0001 complete [ 9347.851415] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_hostid [ 9357.981841] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 9394.426733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9394.643521] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9394.679852] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9394.759796] Lustre: lustre-MDT0000: new disk, initializing [ 9394.879299] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9394.882459] Lustre: Skipped 1 previous similar message [ 9394.908611] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9398.670672] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 9407.806844] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9407.894395] Lustre: 241168:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9407.950240] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9407.953710] Lustre: Skipped 1 previous similar message [ 9408.037794] Lustre: lustre-MDT0001: new disk, initializing [ 9408.196869] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9408.215436] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9411.795324] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 9420.835532] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9425.966653] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9433.695744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9433.908740] Lustre: lustre-OST0000: new disk, initializing [ 9433.913259] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9433.920758] Lustre: 243136:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9434.009906] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9434.014555] Lustre: Skipped 1 previous similar message [ 9435.488907] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9435.499375] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9435.654111] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9440.994751] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 9450.655975] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9454.696825] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9454.841713] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9459.090703] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9459.235662] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9461.216128] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9461.220150] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9461.223986] LustreError: Skipped 1 previous similar message [ 9461.227442] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9461.234077] Lustre: Skipped 1 previous similar message [ 9475.039176] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9475.226628] Lustre: server umount lustre-OST0000 complete [ 9480.162672] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9480.166396] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9485.546184] Lustre: server umount lustre-MDT0000 complete [ 9486.816327] LustreError: 241178:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9486.844082] LustreError: 241178:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9490.491319] LustreError: 241159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450541 with bad export cookie 8636047081621795383 [ 9490.500285] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9490.503620] LustreError: 241159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9490.836930] Lustre: server umount lustre-MDT0001 complete [ 9500.536932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9501.147808] LustreError: 245351:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9501.292968] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9506.200562] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 9514.282967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9519.305597] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 9525.442620] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9530.771929] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9537.539800] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9543.229764] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 9551.059570] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9698.792493] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9698.801902] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9698.811565] Lustre: Skipped 1 previous similar message [ 9698.817055] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9698.819095] Lustre: Skipped 8 previous similar messages [ 9703.299338] Lustre: server umount lustre-OST0000 complete [ 9709.539633] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9709.547807] LustreError: Skipped 1 previous similar message [ 9713.659913] Lustre: server umount lustre-MDT0000 complete [ 9716.193844] LustreError: 246080:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9716.223377] LustreError: 246080:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9717.360740] LustreError: 245331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450768 with bad export cookie 8636047081621796265 [ 9717.369263] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9717.377763] LustreError: 245331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9717.675839] Lustre: server umount lustre-MDT0001 complete [ 9821.485554] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 9824.539669] Key type lgssc unregistered [ 9824.834346] LNet: 248962:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9824.837992] LNetError: 248962:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9825.896901] LNet: Removed LNI 192.168.201.160@tcp [ 9827.007145] Key type .llcrypt unregistered [ 9827.014074] Key type ._llcrypt unregistered [ 9856.122568] Key type ._llcrypt registered [ 9856.124850] Key type .llcrypt registered [ 9856.216608] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_hostid [ 9872.660264] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 9873.652514] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9873.688492] alg: No test for adler32 (adler32-zlib) [ 9874.690239] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 9874.885611] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [ 9876.577356] Key type lgssc registered [ 9877.550208] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9923.329042] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 9932.760681] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9932.780590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9934.064237] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9934.090879] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9934.169700] Lustre: lustre-MDT0000: new disk, initializing [ 9934.237310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9934.250349] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9938.133698] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 9947.562642] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9947.644181] Lustre: 253030:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9947.681821] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9947.684914] Lustre: Skipped 1 previous similar message [ 9947.736685] Lustre: lustre-MDT0001: new disk, initializing [ 9947.816377] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9947.851705] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9947.870597] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9951.678476] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 9956.361340] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9962.138627] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9962.373427] Lustre: lustre-OST0000: new disk, initializing [ 9962.377284] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9962.380985] Lustre: 254661:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9962.444794] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9963.507042] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9963.513688] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9963.604749] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9967.995765] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 9977.011104] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9977.180434] Lustre: lustre-OST0001: new disk, initializing [ 9977.188269] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9977.197226] Lustre: 255532:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9977.273214] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9983.353378] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 9984.607918] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9984.616884] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9984.731859] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9993.127148] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10003.555610] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10015.718261] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10015.736428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10015.746482] Lustre: Skipped 3 previous similar messages [10015.776261] Lustre: Skipped 3 previous similar messages [10020.841284] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10020.857343] Lustre: Skipped 3 previous similar messages [10021.580566] Lustre: server umount lustre-MDT0000 complete [10025.589022] LustreError: 256302:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451076 with bad export cookie 11607976524438819293 [10025.592906] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10025.612714] LustreError: 256302:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10026.001248] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10026.246113] Lustre: server umount lustre-MDT0001 complete [10032.711657] Lustre: server umount lustre-OST0000 complete [10037.434375] Lustre: server umount lustre-OST0001 complete [10053.057279] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10062.885419] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10072.036945] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10081.786423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10101.909441] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [10113.539050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10113.552888] Lustre: lustre-MDT0000: reset Object Index mappings [10114.024435] LustreError: 260545:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10114.043778] LustreError: 260545:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [10114.095190] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10119.151614] LustreError: 260546:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10120.417279] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [10123.251848] LustreError: 260545:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10128.361893] LustreError: 260546:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10129.352937] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10129.400832] Lustre: lustre-MDT0001: reset Object Index mappings [10129.971224] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10134.255950] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [10136.911299] Lustre: 261687:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10143.068147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10148.258591] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [10154.600659] LustreError: 262039:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10156.123503] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10156.360969] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10156.371257] Lustre: Skipped 1 previous similar message [10159.409830] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10159.414203] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10161.934931] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [10168.766962] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10172.592290] Lustre: 263558:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10180.068857] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10180.070476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10180.085942] Lustre: Skipped 4 previous similar messages [10180.089968] Lustre: Skipped 2 previous similar messages [10183.115258] Lustre: server umount lustre-MDT0000 complete [10185.190725] LustreError: 260542:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10185.214584] LustreError: 260542:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10186.378428] LustreError: 263575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451237 with bad export cookie 11607976524438826083 [10186.382085] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10186.390244] LustreError: 263575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10186.789298] Lustre: server umount lustre-MDT0001 complete [10200.696521] Lustre: server umount lustre-OST0000 complete [10215.807695] Lustre: server umount lustre-OST0001 complete [10240.536056] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 12:01:29 (1788451289) [10259.033901] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_hostid [10271.233712] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [10310.622949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10310.877604] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10310.934942] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10311.058519] Lustre: lustre-MDT0000: new disk, initializing [10311.158905] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10311.189835] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10315.966605] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [10327.128722] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10327.331426] Lustre: 267828:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10327.409660] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10327.426574] Lustre: Skipped 1 previous similar message [10327.508938] Lustre: lustre-MDT0001: new disk, initializing [10327.639468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10327.645106] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10331.711401] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [10341.357210] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10347.065934] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10354.106209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10354.407769] Lustre: lustre-OST0000: new disk, initializing [10354.411591] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10354.418592] Lustre: 269797:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10356.233941] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10356.239177] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10356.412650] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10360.035382] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [10368.625768] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10373.658342] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10373.849397] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10378.087297] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10378.299905] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10381.792092] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10381.795871] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10381.821459] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10381.835356] Lustre: Skipped 1 previous similar message [10382.315619] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10386.230523] Lustre: server umount lustre-OST0000 complete [10392.545562] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10392.558841] Lustre: Skipped 2 previous similar messages [10392.560037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10392.580414] Lustre: Skipped 3 previous similar messages [10402.787024] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10402.802685] Lustre: Skipped 3 previous similar messages [10404.319123] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10404.544057] Lustre: server umount lustre-MDT0000 complete [10407.830959] LustreError: 267819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451458 with bad export cookie 11607976524438828176 [10407.838825] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10407.843714] LustreError: 267819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10408.264492] Lustre: server umount lustre-MDT0001 complete [10426.016856] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [10435.504631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10435.865861] LustreError: 272732:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10435.900112] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10435.907297] Lustre: Skipped 2 previous similar messages [10440.665808] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [10449.920071] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10454.913499] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [10458.310234] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10465.579351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10472.265519] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [10481.827223] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10482.008283] Lustre: lustre-OST0001: new disk, initializing [10482.015645] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10482.023160] Lustre: 275039:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10484.028815] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10484.035586] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10484.149112] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10488.196310] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [10499.397621] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10503.704950] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10509.398966] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10509.473753] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10516.961816] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10516.973430] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10517.015578] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10530.271412] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10530.755197] Lustre: server umount lustre-MDT0000 complete [10535.020287] LustreError: 272714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451585 with bad export cookie 11607976524438829058 [10535.033383] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10535.399169] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10535.401392] LustreError: 276113:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10535.423396] Lustre: Skipped 3 previous similar messages [10535.451546] LustreError: 276113:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10535.603406] Lustre: server umount lustre-MDT0001 complete [10542.078809] Lustre: server umount lustre-OST0000 complete [10546.042977] Lustre: server umount lustre-OST0001 complete [10563.122669] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [10573.820476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10574.695768] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10574.709506] Lustre: Skipped 3 previous similar messages [10579.602243] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [10588.222962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10594.095757] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [10597.164802] Lustre: 279237:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10604.082366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10611.221358] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [10613.681619] LustreError: 279590:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10613.697447] LustreError: 279590:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [10619.897204] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10620.648053] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10626.065604] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10627.621534] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [10637.164653] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10641.050592] Lustre: 281112:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10646.403341] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10646.441260] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10657.475637] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 12:08:27 (1788451707) [10675.682035] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10675.697398] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10675.718294] Lustre: Skipped 1 previous similar message [10675.733549] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10675.743247] Lustre: Skipped 12 previous similar messages [10679.702094] Lustre: server umount lustre-MDT0000 complete [10683.657568] LustreError: 278077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451734 with bad export cookie 11607976524438835638 [10683.660704] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10683.673775] LustreError: 278077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [10690.204226] Lustre: server umount lustre-MDT0001 complete [10700.367709] Lustre: server umount lustre-OST0000 complete [10705.782566] Lustre: server umount lustre-OST0001 complete [10714.931985] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_hostid [10725.455710] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [10774.719116] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [10785.961399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10786.205591] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10786.230055] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10786.316053] Lustre: lustre-MDT0000: new disk, initializing [10786.416445] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10792.156864] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [10804.017665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10804.174949] Lustre: 285359:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10804.482926] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10809.417965] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [10822.454428] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10822.716407] Lustre: 286994:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10824.274603] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10830.227339] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [10841.413759] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10841.609204] Lustre: 287865:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10841.696289] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10841.707893] Lustre: Skipped 6 previous similar messages [10842.884490] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10842.897807] Lustre: Skipped 1 previous similar message [10843.081584] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10848.851396] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [10859.781078] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10864.602997] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10864.605931] Lustre: Skipped 1 previous similar message [10880.382280] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 12:12:10 (1788451930) [10884.067485] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10884.077509] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10884.104805] Lustre: Skipped 6 previous similar messages [10884.124673] Lustre: Skipped 11 previous similar messages [10884.950703] Lustre: server umount lustre-MDT0000 complete [10888.622806] LustreError: 285352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451939 with bad export cookie 11607976524438843303 [10888.629982] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10888.646894] LustreError: 285352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10889.043567] Lustre: server umount lustre-MDT0001 complete [10904.751743] Lustre: server umount lustre-OST0000 complete [10905.631150] Lustre: 250540:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788451940/real 1788451940] req@ffff8d4cc21b3800 x1875326717520128/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788451956 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10909.036161] Lustre: server umount lustre-OST0001 complete [10917.757824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10918.169139] LustreError: 290112:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10918.204858] LustreError: 290112:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [10923.320677] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [10932.202934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10937.432315] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [10945.228992] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10951.415781] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10958.449202] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10966.777314] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [10976.159497] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10996.561767] loop1: detected capacity change from 0 to 204800000 [10996.914561] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10996.928712] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10996.938995] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11002.524896] loop1: detected capacity change from 0 to 204800000 [11008.358634] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11008.432484] LustreError: 290100:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs14097: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11008.442642] LustreError: 290100:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs14097 [11008.446726] LustreError: 290100:0:(mgs_handler.c:517:mgs_target_reg()) fs14097-MDT0001: Error checking target index: rc = -13 [11008.452468] LustreError: fs14097-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11008.458687] LustreError: 293834:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11008.465539] LustreError: 293834:0:(tgt_mount.c:2129:server_put_super()) no obd fs14097-MDT0001 [11008.470534] LustreError: 293834:0:(tgt_mount.c:116:server_deregister_mount()) fs14097-MDT0001 not registered [11008.530517] Lustre: server umount fs14097-MDT0001 complete [11008.533503] LustreError: 293834:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [11017.165178] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11017.448933] Lustre: Setting parameter fs14097-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs14097-MDT0001 [11017.539954] Lustre: srv-fs14097-MDT0001: No data found on store. Initialize space. [11017.544779] Lustre: Skipped 4 previous similar messages [11017.646567] Lustre: fs14097-MDT0001: new disk, initializing [11017.653444] Lustre: Skipped 3 previous similar messages [11017.855779] Lustre: srv-fs14097-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11017.868763] Lustre: 294300:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs14097-MDT0001-mdtlov: can't create new llog: rc = -115 [11017.889153] LustreError: 294300:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs14097-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11022.398867] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [11029.332616] Lustre: server umount fs14097-MDT0001 complete [11043.978617] loop1: detected capacity change from 0 to 204800000 [11044.338534] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11050.321537] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11050.716613] Lustre: 295869:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs14097-MDT0001-mdtlov: can't create new llog: rc = -115 [11050.720733] LustreError: 295869:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs14097-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11055.677679] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [11059.693641] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11060.879484] Lustre: server umount fs14097-MDT0001 complete [11074.311946] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.60@tcp (stopping) [11075.053377] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11075.086050] Lustre: Skipped 5 previous similar messages [11077.786640] Lustre: server umount lustre-MDT0000 complete [11087.683759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11087.903831] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11093.500229] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11093.524670] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [11101.739895] loop1: detected capacity change from 0 to 204800000 [11108.181187] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11108.762958] Lustre: 298316:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs14097-MDT0001-mdtlov: can't create new llog: rc = -115 [11108.781167] LustreError: 298316:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs14097-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11113.518989] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [11118.903558] Lustre: server umount fs14097-MDT0001 complete [11141.600589] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11154.400085] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11154.878756] Lustre: server umount lustre-OST0000 complete [11168.137921] Lustre: server umount lustre-MDT0000 complete [11172.979526] LustreError: 290857:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452223 with bad export cookie 11607976524438845984 [11172.983286] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11172.989249] LustreError: 290857:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11173.361659] Lustre: server umount lustre-MDT0001 complete [11186.866333] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [11189.964466] Key type lgssc unregistered [11190.373849] LNet: 300665:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11190.386426] LNetError: 300665:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11191.398632] LNet: Removed LNI 192.168.201.160@tcp [11192.575392] Key type .llcrypt unregistered [11192.586022] Key type ._llcrypt unregistered [11215.163671] Key type ._llcrypt registered [11215.166358] Key type .llcrypt registered [11215.277554] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [11216.754609] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11216.795678] alg: No test for adler32 (adler32-zlib) [11217.839415] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [11218.118746] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [11219.833449] Key type lgssc registered [11221.048661] Lustre: Echo OBD driver; http://www.lustre.org/ [11229.760712] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11229.809044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11231.299727] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11235.635402] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [11241.447456] LustreError: 301952:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11241.469848] LustreError: 301952:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11244.916460] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11245.278351] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11250.624437] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [11259.553654] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11266.541196] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11274.717822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11275.070034] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11277.113277] LustreError: 303784:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11282.426257] LustreError: 303782:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11283.230833] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [11287.530929] LustreError: 303785:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11291.847668] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11293.918921] LustreError: 303784:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11293.941553] LustreError: 303784:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11304.173116] LustreError: 303782:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11304.247789] LustreError: 303782:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [11316.874119] loop1: detected capacity change from 0 to 204800000 [11317.166435] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11317.182863] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11317.220952] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11321.333935] LustreError: 303782:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11321.358992] LustreError: 303782:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [11323.009243] loop1: detected capacity change from 0 to 204800000 [11329.296872] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11329.509817] Lustre: fs3439-OST0000: new disk, initializing [11329.512890] Lustre: srv-fs3439-OST0000: No data found on store. Initialize space. [11329.518594] Lustre: 305689:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11329.567641] Lustre: fs3439-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11336.397782] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [11344.187028] Lustre: server umount fs3439-OST0000 complete [11355.369469] LustreError: 303784:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11355.389952] LustreError: 303784:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 18 previous similar messages [11357.174187] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11357.439668] loop1: detected capacity change from 0 to 204800000 [11362.799552] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11363.023459] Lustre: fs3439-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11370.497727] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [11372.529843] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11379.595068] loop2: detected capacity change from 0 to 204800000 [11379.783498] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11379.799230] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11379.815829] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11384.672685] loop2: detected capacity change from 0 to 204800000 [11390.564654] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11390.712986] LustreError: fs3439-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11390.730696] LustreError: fs3439-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11390.742771] LustreError: 308728:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11390.757565] LustreError: 308728:0:(tgt_mount.c:2129:server_put_super()) no obd fs3439-OST0001 [11390.768850] LustreError: 308728:0:(tgt_mount.c:116:server_deregister_mount()) fs3439-OST0001 not registered [11390.828472] Lustre: server umount fs3439-OST0001 complete [11390.836787] LustreError: 308728:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [11405.332052] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11405.779668] Lustre: fs3439-OST0001: new disk, initializing [11405.782955] Lustre: srv-fs3439-OST0001: No data found on store. Initialize space. [11405.801465] Lustre: 309280:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11405.948704] Lustre: fs3439-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11415.374149] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [11421.927501] LustreError: 303784:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11421.954903] LustreError: 303784:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 38 previous similar messages [11425.119775] Lustre: server umount fs3439-OST0000 complete [11436.356791] Lustre: server umount fs3439-OST0001 complete [11449.949645] loop1: detected capacity change from 0 to 204800000 [11456.417174] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11456.693561] Lustre: fs3439-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11463.456909] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [11470.645308] loop2: detected capacity change from 0 to 204800000 [11476.719228] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11484.112396] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [11493.812216] Lustre: server umount fs3439-OST0000 complete [11504.518514] Lustre: server umount fs3439-OST0001 complete [11510.766373] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11520.994333] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11521.001557] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11521.031095] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11521.046830] Lustre: Skipped 1 previous similar message [11525.826317] Lustre: server umount lustre-OST0000 complete [11531.234540] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11531.259666] Lustre: Skipped 1 previous similar message [11531.267056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11536.307508] Lustre: server umount lustre-MDT0000 complete [11540.833529] LustreError: 301934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452591 with bad export cookie 3851935788091210426 [11540.841973] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11540.848292] LustreError: 301934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11541.293795] Lustre: server umount lustre-MDT0001 complete [11553.841110] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [11556.778725] Key type lgssc unregistered [11557.131764] LNet: 315208:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11557.146158] LNetError: 315208:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11557.173174] LNet: Removed LNI 192.168.201.160@tcp [11558.062173] Key type .llcrypt unregistered [11558.063699] Key type ._llcrypt unregistered [11594.842978] Key type ._llcrypt registered [11594.846280] Key type .llcrypt registered [11594.937144] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_hostid [11613.168479] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [11615.218520] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11615.305473] alg: No test for adler32 (adler32-zlib) [11616.619275] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [11616.941248] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [11618.680672] Key type lgssc registered [11620.172706] Lustre: Echo OBD driver; http://www.lustre.org/ [11663.156678] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11663.187966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11664.547696] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11664.592986] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11664.741310] Lustre: lustre-MDT0000: new disk, initializing [11664.845300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11664.864335] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11669.250516] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [11680.659024] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11680.855768] Lustre: 319072:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11680.935708] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11680.955327] Lustre: Skipped 1 previous similar message [11681.113600] Lustre: lustre-MDT0001: new disk, initializing [11681.259872] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11681.306132] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11681.327349] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11686.018873] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [11696.095771] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11701.447488] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11707.593763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11707.950429] Lustre: lustre-OST0000: new disk, initializing [11707.955434] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11707.963583] Lustre: 321040:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11708.085329] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11714.115060] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11714.127636] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11714.290754] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11714.470306] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [11725.383582] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11727.677582] Lustre: Mounted lustre-client - version 2.17.56_51_g2aa7cea [11728.799256] Lustre: Unmounted lustre-client [11740.129747] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11740.135701] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11740.148478] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11740.155416] Lustre: Skipped 1 previous similar message [11743.204972] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11743.221259] Lustre: Skipped 1 previous similar message [11746.206824] Lustre: server umount lustre-OST0000 complete [11752.931736] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11752.944563] LustreError: Skipped 1 previous similar message [11752.956707] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11752.976704] Lustre: Skipped 1 previous similar message [11752.993916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11756.545611] Lustre: server umount lustre-MDT0000 complete [11761.154190] LustreError: 321039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452812 with bad export cookie 9207686032011998848 [11761.156306] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11761.161272] LustreError: 321039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11761.408739] Lustre: server umount lustre-MDT0001 complete [11775.350353] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [11778.506169] Key type lgssc unregistered [11785.951155] LNet: 1 local NIs in recovery (showing 1): 192.168.201.160@tcp [11785.954618] LNet: 323206:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11785.969900] LNetError: 323206:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11859.791926] LNet: Removed LNI 192.168.201.160@tcp [11861.410415] Key type .llcrypt unregistered [11861.412335] Key type ._llcrypt unregistered [11883.418268] Key type ._llcrypt registered [11883.420407] Key type .llcrypt registered [11883.502653] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_hostid [11905.062824] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [11906.681756] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11906.920369] alg: No test for adler32 (adler32-zlib) [11908.067417] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [11908.421572] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [11910.167239] Key type lgssc registered [11911.720870] Lustre: Echo OBD driver; http://www.lustre.org/ [11951.811372] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 12:30:01 (1788453001) [11959.581592] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11959.649294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11960.995482] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11961.033564] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11961.137966] Lustre: lustre-MDT0000: new disk, initializing [11961.199920] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11961.225620] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11966.300826] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [11979.561489] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11979.748302] Lustre: 326434:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11979.846092] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11979.852096] Lustre: Skipped 1 previous similar message [11979.988331] Lustre: lustre-MDT0001: new disk, initializing [11980.090774] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11980.136056] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11980.163616] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11985.150176] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [11995.581974] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12002.242261] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12011.200884] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12011.643746] Lustre: lustre-OST0000: new disk, initializing [12011.646058] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12011.652283] Lustre: 328408:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12011.773229] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12018.482713] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12018.500757] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12018.726198] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12020.250309] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [12030.905215] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12049.381429] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12049.387739] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12049.421886] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12049.434876] Lustre: Skipped 1 previous similar message [12052.237197] Lustre: server umount lustre-OST0000 complete [12057.064196] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12057.072869] LustreError: Skipped 1 previous similar message [12057.085346] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12057.116565] Lustre: Skipped 1 previous similar message [12057.122389] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12059.616469] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12059.618310] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12064.749510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12064.757892] Lustre: Skipped 2 previous similar messages [12069.863949] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12069.874030] Lustre: Skipped 1 previous similar message [12070.367251] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12070.573900] Lustre: server umount lustre-MDT0000 complete [12074.767074] LustreError: 326429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453125 with bad export cookie 18249724673515333695 [12074.772254] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12074.781215] LustreError: 326429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12075.082157] Lustre: server umount lustre-MDT0001 complete [12090.010592] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [12094.148702] Key type lgssc unregistered [12094.505139] LNet: 330321:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12094.518123] LNetError: 330321:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12094.538280] LNet: Removed LNI 192.168.201.160@tcp [12095.891477] Key type .llcrypt unregistered [12095.905135] Key type ._llcrypt unregistered [12126.789351] Key type ._llcrypt registered [12126.791947] Key type .llcrypt registered [12126.952053] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [12128.289253] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12128.398028] alg: No test for adler32 (adler32-zlib) [12129.627449] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [12129.837659] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [12131.511370] Key type lgssc registered [12132.810324] Lustre: Echo OBD driver; http://www.lustre.org/ [12142.904581] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12142.933326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12144.302353] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12149.336860] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [12154.855719] LustreError: 331598:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12154.900746] LustreError: 331598:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12159.586819] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12160.083712] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12165.831973] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [12175.768801] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12182.811366] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12190.988766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12191.582158] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12199.828056] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [12209.017669] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12225.071419] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12227.067421] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12234.722685] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12234.728910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12234.752391] Lustre: Skipped 2 previous similar messages [12234.771855] Lustre: Skipped 2 previous similar messages [12236.549977] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.60@tcp (stopping) [12238.617409] Lustre: server umount lustre-MDT0000 complete [12239.845766] LustreError: 331598:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12239.866077] LustreError: 331598:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12241.655293] LustreError: 331593:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12242.836043] LustreError: 331578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453293 with bad export cookie 16007645662372241841 [12242.860995] LustreError: 331578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12242.863077] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12244.960278] LustreError: 331594:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12244.961784] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12245.008062] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12247.974756] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12247.982101] Lustre: Skipped 1 previous similar message [12251.872448] LustreError: 331593:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12251.900098] LustreError: 331593:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12253.089428] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12253.094869] Lustre: Skipped 1 previous similar message [12257.247213] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12257.432485] Lustre: server umount lustre-MDT0001 complete [12262.245854] Lustre: server umount lustre-OST0000 complete [12271.899714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12277.152305] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [12280.428480] Lustre: 335786:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.160@tcp [12282.880300] Lustre: server umount MGS complete [12292.869270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12293.641822] LustreError: 336359:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12293.671041] LustreError: 336359:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12293.806437] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12299.990136] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [12309.227869] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12309.530275] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12314.738380] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [12324.797990] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12325.343229] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12332.566683] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [12342.169565] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12344.355333] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12351.744306] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12354.567549] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12359.136709] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12359.145254] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12359.158490] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12359.171067] Lustre: Skipped 1 previous similar message [12359.181191] Lustre: Skipped 2 previous similar messages [12365.345315] Lustre: server umount lustre-OST0000 complete [12370.913613] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12370.917160] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12376.535955] Lustre: server umount lustre-MDT0000 complete [12379.620486] LustreError: 336355:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12379.633426] LustreError: 336355:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [12381.263955] LustreError: 336340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453432 with bad export cookie 16007645662372242681 [12381.267676] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12381.295101] LustreError: 336340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [12381.765981] Lustre: server umount lustre-MDT0001 complete [12395.123343] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [12398.362993] Key type lgssc unregistered [12398.702332] LNet: 339799:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12398.713779] LNetError: 339799:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12398.730223] LNet: Removed LNI 192.168.201.160@tcp [12399.835978] Key type .llcrypt unregistered [12399.838921] Key type ._llcrypt unregistered [12422.637060] Key type ._llcrypt registered [12422.639883] Key type .llcrypt registered [12422.766625] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_hostid [12440.093333] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [12440.800377] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12440.817250] alg: No test for adler32 (adler32-zlib) [12441.938934] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [12442.124289] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [12443.767593] Key type lgssc registered [12444.845753] Lustre: Echo OBD driver; http://www.lustre.org/ [12489.497045] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12489.537167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12490.923483] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12490.999437] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12491.090244] Lustre: lustre-MDT0000: new disk, initializing [12491.243502] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12491.308775] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12498.030860] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [12511.586102] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12511.806576] Lustre: 342958:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12511.903900] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12511.912632] Lustre: Skipped 1 previous similar message [12512.178182] Lustre: lustre-MDT0001: new disk, initializing [12512.377936] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12512.455440] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12512.475213] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12518.541632] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [12529.338737] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12536.567508] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12546.732103] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12547.049749] Lustre: lustre-OST0000: new disk, initializing [12547.058360] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12547.076418] Lustre: 344930:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12547.213078] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12553.875812] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12553.906238] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12554.086891] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12556.549342] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [12567.251799] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12572.697794] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12572.878656] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12578.245387] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12578.437146] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12584.940978] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12584.953808] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12584.960259] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12584.968306] Lustre: Skipped 1 previous similar message [12585.003201] Lustre: Skipped 1 previous similar message [12587.009934] Lustre: server umount lustre-OST0000 complete [12594.145800] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12594.151396] LustreError: Skipped 1 previous similar message [12594.155969] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12594.177618] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12595.173477] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12595.196044] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12595.211256] Lustre: Skipped 1 previous similar message [12597.198734] Lustre: server umount lustre-MDT0000 complete [12602.338167] LustreError: 343755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453653 with bad export cookie 11692585882816996993 [12602.348646] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12602.353176] LustreError: 343755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12603.025582] Lustre: server umount lustre-MDT0001 complete [12613.455830] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 12:41:02 (1788453662) [12632.515746] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_hostid [12648.510135] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [12693.736390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12693.951635] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12693.980924] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12694.087598] Lustre: lustre-MDT0000: new disk, initializing [12694.187313] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12694.206682] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12700.258906] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [12713.570052] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12713.736273] Lustre: 349716:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12713.776739] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12713.780702] Lustre: Skipped 1 previous similar message [12713.879030] Lustre: lustre-MDT0001: new disk, initializing [12714.001380] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12714.062333] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12714.090442] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12718.576779] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [12729.299791] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12737.201794] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12745.706312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12745.980572] Lustre: lustre-OST0000: new disk, initializing [12745.986437] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12745.992286] Lustre: 351682:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12746.083253] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12752.791194] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [12754.007136] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12754.021069] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12754.192430] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12762.817450] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12800.487334] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12800.493927] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12800.500249] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12800.520233] Lustre: Skipped 1 previous similar message [12804.780959] Lustre: server umount lustre-OST0000 complete [12810.721915] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12810.731580] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12810.738300] Lustre: Skipped 1 previous similar message [12810.753640] Lustre: Skipped 2 previous similar messages [12815.407671] Lustre: server umount lustre-MDT0000 complete [12815.846959] LustreError: 352564:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12815.871047] LustreError: 352564:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [12819.301600] LustreError: 349707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453870 with bad export cookie 11692585882816997875 [12819.304880] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12819.308268] LustreError: 349707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12819.695906] Lustre: server umount lustre-MDT0001 complete [12831.834683] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [12835.333944] Key type lgssc unregistered [12835.694927] LNet: 353864:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12835.705219] LNetError: 353864:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12835.736069] LNet: Removed LNI 192.168.201.160@tcp [12836.635410] Key type .llcrypt unregistered [12836.637677] Key type ._llcrypt unregistered [12859.018348] Key type ._llcrypt registered [12859.020083] Key type .llcrypt registered [12859.124187] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [12860.174756] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12860.214335] alg: No test for adler32 (adler32-zlib) [12861.272809] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [12861.483205] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [12863.135835] Key type lgssc registered [12864.041279] Lustre: Echo OBD driver; http://www.lustre.org/ [12871.033817] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12871.045897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12872.355265] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12877.565916] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [12882.917961] LustreError: 355136:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12882.948990] LustreError: 355136:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12887.929348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12888.034354] LustreError: 355137:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12888.377982] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12894.052642] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [12903.466319] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12909.209631] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12915.101375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12915.359895] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12920.618900] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [12927.313190] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12943.329947] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12943.333130] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12943.360481] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12944.359955] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12944.375187] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12944.389368] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12949.478965] LustreError: 356960:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12949.653591] Lustre: server umount lustre-OST0000 complete [12954.599397] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12954.602639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12954.623839] Lustre: Skipped 1 previous similar message [12954.638398] Lustre: Skipped 2 previous similar messages [12959.712560] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12959.720727] Lustre: Skipped 1 previous similar message [12960.319899] Lustre: server umount lustre-MDT0000 complete [12964.838153] LustreError: 356081:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12964.860131] LustreError: 356081:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12965.260492] LustreError: 355116:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454016 with bad export cookie 18251823419335552271 [12965.267243] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12965.275345] LustreError: 355116:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12965.947431] Lustre: server umount lustre-MDT0001 complete [12981.584387] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [12986.684979] Key type lgssc unregistered [12987.121687] LNet: 360744:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12987.126248] LNetError: 360744:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12987.147295] LNet: Removed LNI 192.168.201.160@tcp [12988.621633] Key type .llcrypt unregistered [12988.628684] Key type ._llcrypt unregistered [13013.785616] Key type ._llcrypt registered [13013.787484] Key type .llcrypt registered [13013.857795] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [13014.956037] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13014.984192] alg: No test for adler32 (adler32-zlib) [13016.189372] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [13016.477109] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [13018.273957] Key type lgssc registered [13019.345916] Lustre: Echo OBD driver; http://www.lustre.org/ [13028.391833] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13028.431774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13029.910551] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13034.789508] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [13040.101855] LustreError: 365004:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13040.114068] LustreError: 365004:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13045.219941] LustreError: 365002:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13045.807929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13046.234935] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13051.743218] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [13061.436901] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13069.067511] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13079.136722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13079.579549] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13088.380410] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [13098.077962] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13116.386930] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13116.389586] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13116.392584] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13116.420749] Lustre: Skipped 1 previous similar message [13116.439645] Lustre: Skipped 1 previous similar message [13119.118531] Lustre: server umount lustre-OST0000 complete [13126.628387] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13126.635814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13126.642809] Lustre: Skipped 1 previous similar message [13126.654420] Lustre: Skipped 1 previous similar message [13130.441471] Lustre: server umount lustre-MDT0000 complete [13131.757434] LustreError: 365002:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13131.778100] LustreError: 365002:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13134.886435] LustreError: 364921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454185 with bad export cookie 11912749545777728519 [13134.905595] LustreError: 364921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13134.906227] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13135.326776] Lustre: server umount lustre-MDT0001 complete [13147.821484] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [13150.998210] Key type lgssc unregistered [13151.415450] LNet: 372795:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13151.427420] LNetError: 372795:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13151.456656] LNet: Removed LNI 192.168.201.160@tcp [13152.491691] Key type .llcrypt unregistered [13152.494345] Key type ._llcrypt unregistered [13176.007299] Key type ._llcrypt registered [13176.009800] Key type .llcrypt registered [13176.168623] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [13177.581307] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13177.666775] alg: No test for adler32 (adler32-zlib) [13178.906768] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [13179.202100] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [13180.921384] Key type lgssc registered [13183.049915] Lustre: Echo OBD driver; http://www.lustre.org/ [13195.097798] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13195.148580] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13197.099139] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13202.240940] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [13206.505512] LustreError: 376757:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13206.535957] LustreError: 376757:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13211.630666] LustreError: 376759:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13212.840513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13213.229988] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13218.447426] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [13227.272611] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13234.491634] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13242.676580] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13243.050774] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13250.684952] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [13259.926840] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13282.273808] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13282.278810] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13282.294893] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13282.307615] Lustre: Skipped 1 previous similar message [13287.297528] Lustre: server umount lustre-OST0000 complete [13292.518722] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13292.524623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13292.534534] Lustre: Skipped 1 previous similar message [13292.553818] Lustre: Skipped 2 previous similar messages [13297.632727] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13297.640976] Lustre: Skipped 1 previous similar message [13297.967618] Lustre: server umount lustre-MDT0000 complete [13301.762749] LustreError: 380207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454352 with bad export cookie 2056574378871168849 [13301.765188] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13301.773397] LustreError: 380207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13302.200949] Lustre: server umount lustre-MDT0001 complete [13314.912143] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [13317.748394] Key type lgssc unregistered [13318.085177] LNet: 383024:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13318.093450] LNetError: 383024:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13318.107596] LNet: Removed LNI 192.168.201.160@tcp [13319.326235] Key type .llcrypt unregistered [13319.328058] Key type ._llcrypt unregistered [13423.329920] Key type ._llcrypt registered [13423.332804] Key type .llcrypt registered [13423.473476] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [13423.837459] Key type .llcrypt unregistered [13423.839086] Key type ._llcrypt unregistered [13509.659968] Key type ._llcrypt registered [13509.661866] Key type .llcrypt registered [13509.777667] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [13510.113368] Key type .llcrypt unregistered [13510.115397] Key type ._llcrypt unregistered [13542.553701] Key type ._llcrypt registered [13542.555689] Key type .llcrypt registered [13542.682232] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [13543.040424] Key type .llcrypt unregistered [13543.042124] Key type ._llcrypt unregistered [13578.463896] Key type ._llcrypt registered [13578.466705] Key type .llcrypt registered [13578.598694] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [13579.122424] Key type .llcrypt unregistered [13579.123994] Key type ._llcrypt unregistered [13665.719134] Key type ._llcrypt registered [13665.722409] Key type .llcrypt registered [13665.875798] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [13666.284520] Key type .llcrypt unregistered [13666.288996] Key type ._llcrypt unregistered [13699.024698] Key type ._llcrypt registered [13699.026815] Key type .llcrypt registered [13699.144830] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [13699.440562] Key type .llcrypt unregistered [13699.443904] Key type ._llcrypt unregistered [13733.405824] Key type ._llcrypt registered [13733.412472] Key type .llcrypt registered [13733.554187] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [13733.975699] Key type .llcrypt unregistered [13733.977372] Key type ._llcrypt unregistered [13768.338929] Key type ._llcrypt registered [13768.341392] Key type .llcrypt registered [13768.437924] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [13768.962336] Key type .llcrypt unregistered [13768.966373] Key type ._llcrypt unregistered [13800.201036] Key type ._llcrypt registered [13800.203841] Key type .llcrypt registered [13800.307736] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [13800.591964] Key type .llcrypt unregistered [13800.596832] Key type ._llcrypt unregistered [13834.384754] Key type ._llcrypt registered [13834.386855] Key type .llcrypt registered [13834.499884] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [13834.753497] Key type .llcrypt unregistered [13834.756279] Key type ._llcrypt unregistered [13913.137734] Key type ._llcrypt registered [13913.139249] Key type .llcrypt registered [13913.357717] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [13913.877602] Key type .llcrypt unregistered [13913.889421] Key type ._llcrypt unregistered [13953.320851] Key type ._llcrypt registered [13953.323580] Key type .llcrypt registered [13953.562305] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [13953.987503] Key type .llcrypt unregistered [13953.993099] Key type ._llcrypt unregistered [13995.369780] Key type ._llcrypt registered [13995.372574] Key type .llcrypt registered [13995.526720] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [13995.879306] Key type .llcrypt unregistered [13995.883106] Key type ._llcrypt unregistered [14062.300580] Key type ._llcrypt registered [14062.304692] Key type .llcrypt registered [14062.458293] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_hostid [14084.447418] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [14085.362326] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14085.429577] alg: No test for adler32 (adler32-zlib) [14086.562870] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [14086.842531] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [14088.567202] Key type lgssc registered [14090.202715] Lustre: Echo OBD driver; http://www.lustre.org/ [14137.761490] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14137.787420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14139.071285] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [14139.107107] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [14139.180257] Lustre: tst250fs-MDT0000: new disk, initializing [14139.242972] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14139.264435] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14144.445884] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [14156.794639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14156.927834] Lustre: 403694:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14156.994967] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [14156.999619] Lustre: Skipped 1 previous similar message [14157.068362] Lustre: tst250fs-MDT0001: new disk, initializing [14157.174837] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14157.222225] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14157.239208] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14161.980180] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [14167.723674] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [14175.602966] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14176.038694] Lustre: tst250fs-OST0000: new disk, initializing [14176.048090] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [14176.057835] Lustre: 405327:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14176.235401] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14179.762455] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14179.774462] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14179.905305] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14185.662786] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [14199.773782] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14200.052470] Lustre: tst250fs-OST0001: new disk, initializing [14200.060241] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [14200.076636] Lustre: 406199:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14200.214150] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [14207.339945] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [14208.677561] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [14208.691329] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [14208.814704] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [14213.850614] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [14215.939848] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [14215.944761] Lustre: Skipped 1 previous similar message [14239.200482] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14239.221603] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14239.228523] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14244.835437] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14244.850841] Lustre: Skipped 6 previous similar messages [14245.317396] Lustre: server umount tst250fs-MDT0000 complete [14251.350367] LustreError: 403684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788455302 with bad export cookie 6253071759338762320 [14251.352572] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14251.375801] LustreError: 403684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14252.180839] Lustre: server umount tst250fs-MDT0001 complete [14268.903634] Lustre: server umount tst250fs-OST0000 complete [14270.561186] Lustre: 401914:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788455305/real 1788455306] req@ffff8d4e01ae8e00 x1875331133807232/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788455321 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14270.620566] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14270.649741] Lustre: Skipped 3 previous similar messages [14272.294559] Lustre: 401917:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788455307/real 1788455307] req@ffff8d4ccabdd500 x1875331133807488/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788455323 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14274.271725] Lustre: server umount tst250fs-OST0001 complete [14295.802198] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_hostid [14306.382669] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [14345.639219] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 13:09:55 (1788455395) [14347.948106] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [14367.183296] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_hostid [14386.966406] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [14430.400886] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14127 sec ========= 13:11:19 (1788455479) [14433.090170] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 13:11:22 (1788455482) === [14439.092657] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 13:11:28 (1788455488) === [14490.439106] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [14494.513301] Key type lgssc unregistered [14494.950747] LNet: 415375:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14494.955983] LNetError: 415375:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14494.987600] LNet: Removed LNI 192.168.201.160@tcp [14495.993150] Key type .llcrypt unregistered [14495.994997] Key type ._llcrypt unregistered