[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 489940959 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003201] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005016] kvm-guest: setup PV IPIs [ 0.008334] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009026] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010013] pid_max: default: 32768 minimum: 301 [ 0.011142] LSM: Security Framework initializing [ 0.012055] Yama: becoming mindful. [ 0.013067] SELinux: Initializing. [ 0.014066] *** VALIDATE selinux *** [ 0.022817] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027315] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028187] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029135] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030117] *** VALIDATE tmpfs *** [ 0.031519] *** VALIDATE proc *** [ 0.033071] *** VALIDATE cgroup *** [ 0.034010] *** VALIDATE cgroup2 *** [ 0.035285] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036171] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039029] Spectre V2 : User space: Vulnerable [ 0.040015] Speculative Store Bypass: Vulnerable [ 0.044177] debug: unmapping init [mem 0xffffffffbe059000-0xffffffffbe060fff] [ 0.046931] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047653] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048028] ... version: 2 [ 0.049015] ... bit width: 48 [ 0.050012] ... generic registers: 4 [ 0.051012] ... value mask: 0000ffffffffffff [ 0.052015] ... max period: 00007fffffffffff [ 0.053016] ... fixed-purpose events: 3 [ 0.054015] ... event mask: 000000070000000f [ 0.055276] rcu: Hierarchical SRCU implementation. [ 0.057534] smp: Bringing up secondary CPUs ... [ 0.058624] x86: Booting SMP configuration: [ 0.059024] .... node #0, CPUs: #1 #2 #3 [ 0.063013] smp: Brought up 1 node, 4 CPUs [ 0.065017] smpboot: Max logical packages: 1 [ 0.066020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.147194] node 0 deferred pages initialised in 78ms [ 0.150034] devtmpfs: initialized [ 0.151290] x86/mm: Memory block size: 128MB [ 0.154034] gcov: version magic: 0x41383552 [ 0.156250] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.157076] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.158285] pinctrl core: initialized pinctrl subsystem [ 0.159240] [ 0.159913] ************************************************************* [ 0.160018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.161022] ** ** [ 0.162014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.163013] ** ** [ 0.164014] ** This means that this kernel is built to expose internal ** [ 0.165018] ** IOMMU data structures, which may compromise security on ** [ 0.166016] ** your system. ** [ 0.167020] ** ** [ 0.168017] ** If you see this message and you are not debugging the ** [ 0.169011] ** kernel, report this immediately to your vendor! ** [ 0.170015] ** ** [ 0.171016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.172013] ************************************************************* [ 0.173982] NET: Registered protocol family 16 [ 0.174408] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.175069] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.176080] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.177497] cpuidle: using governor menu [ 0.181039] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.184600] PCI: Using configuration type 1 for base access [ 0.186122] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.198170] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.199027] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.201147] cryptd: max_cpu_qlen set to 1000 [ 0.203297] ACPI: Added _OSI(Module Device) [ 0.205021] ACPI: Added _OSI(Processor Device) [ 0.207015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.209017] ACPI: Added _OSI(Processor Aggregator Device) [ 0.214922] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.224606] ACPI: Interpreter enabled [ 0.227077] ACPI: PM: (supports S0 S3 S4 S5) [ 0.229018] ACPI: Using IOAPIC for interrupt routing [ 0.231179] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.235376] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.246434] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.248053] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.250025] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.254088] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.259467] acpiphp: Slot [2] registered [ 0.261115] acpiphp: Slot [5] registered [ 0.263157] acpiphp: Slot [6] registered [ 0.265129] acpiphp: Slot [7] registered [ 0.266104] acpiphp: Slot [8] registered [ 0.268122] acpiphp: Slot [9] registered [ 0.269119] acpiphp: Slot [10] registered [ 0.271174] acpiphp: Slot [3] registered [ 0.273132] acpiphp: Slot [4] registered [ 0.275212] acpiphp: Slot [11] registered [ 0.277150] acpiphp: Slot [12] registered [ 0.278204] acpiphp: Slot [13] registered [ 0.280116] acpiphp: Slot [14] registered [ 0.282090] acpiphp: Slot [15] registered [ 0.284154] acpiphp: Slot [16] registered [ 0.286140] acpiphp: Slot [17] registered [ 0.288171] acpiphp: Slot [18] registered [ 0.289125] acpiphp: Slot [19] registered [ 0.291200] acpiphp: Slot [20] registered [ 0.293089] acpiphp: Slot [21] registered [ 0.295144] acpiphp: Slot [22] registered [ 0.297123] acpiphp: Slot [23] registered [ 0.298198] acpiphp: Slot [24] registered [ 0.300132] acpiphp: Slot [25] registered [ 0.302318] acpiphp: Slot [26] registered [ 0.304140] acpiphp: Slot [27] registered [ 0.306193] acpiphp: Slot [28] registered [ 0.309198] acpiphp: Slot [29] registered [ 0.311113] acpiphp: Slot [30] registered [ 0.312120] acpiphp: Slot [31] registered [ 0.314078] PCI host bridge to bus 0000:00 [ 0.315026] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.318026] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.320022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.323032] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.326041] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.329031] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.331290] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.335069] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.338507] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.348733] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.354485] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.357031] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.360021] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.363025] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.366378] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.369842] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.373054] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.376849] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.382017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.397025] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.402980] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.408992] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.416017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.424023] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.445028] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.455031] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.461016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.468015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.494020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.506032] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.512032] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.517027] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.535024] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.546369] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.552017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.560016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.584027] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.599459] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.608022] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.617023] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.646025] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.664209] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.672025] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.680019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.699024] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.711368] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.713571] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.717546] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.720588] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.725059] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.730151] iommu: Default domain type: Passthrough [ 0.732608] SCSI subsystem initialized [ 0.734165] ACPI: bus type USB registered [ 0.736137] usbcore: registered new interface driver usbfs [ 0.738112] usbcore: registered new interface driver hub [ 0.740108] usbcore: registered new device driver usb [ 0.742224] pps_core: LinuxPPS API ver. 1 registered [ 0.745013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.748095] PTP clock support registered [ 0.751117] EDAC MC: Ver: 3.0.0 [ 0.753144] PCI: Using ACPI for IRQ routing [ 0.754604] NetLabel: Initializing [ 0.756018] NetLabel: domain hash size = 128 [ 0.758013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.760101] NetLabel: unlabeled traffic allowed by default [ 0.763066] vgaarb: loaded [ 0.764394] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.766026] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.779000] clocksource: Switched to clocksource kvm-clock [ 0.895168] VFS: Disk quotas dquot_6.6.0 [ 0.897327] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.900145] *** VALIDATE ramfs *** [ 0.901575] *** VALIDATE hugetlbfs *** [ 0.903530] pnp: PnP ACPI init [ 0.905746] pnp: PnP ACPI: found 6 devices [ 0.925989] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.929812] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.932203] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.934846] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.937724] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.940546] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.944062] NET: Registered protocol family 2 [ 0.947089] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.953342] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.957508] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.964246] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.969779] TCP: Hash tables configured (established 65536 bind 65536) [ 0.974096] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.978647] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.982064] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.985741] NET: Registered protocol family 1 [ 0.988756] RPC: Registered named UNIX socket transport module. [ 0.991392] RPC: Registered udp transport module. [ 0.993353] RPC: Registered tcp transport module. [ 0.995371] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.998287] NET: Registered protocol family 44 [ 1.000323] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.002426] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.004579] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.006918] PCI: CLS 0 bytes, default 64 [ 1.009642] Unpacking initramfs... [ 2.440177] debug: unmapping init [mem 0xffff93cf3cc54000-0xffff93cf3ffbffff] [ 2.444563] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.448246] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.451679] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.013778] Initialise system trusted keyrings [ 3.015393] Key type blacklist registered [ 3.017667] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.029180] zbud: loaded [ 3.035320] *** VALIDATE nfs *** [ 3.036744] *** VALIDATE nfs4 *** [ 3.038413] pstore: using deflate compression [ 3.041820] Platform Keyring initialized [ 3.157558] NET: Registered protocol family 38 [ 3.159357] Key type asymmetric registered [ 3.160468] Asymmetric key parser 'x509' registered [ 3.162072] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.165490] io scheduler mq-deadline registered [ 3.167273] io scheduler kyber registered [ 3.168972] io scheduler bfq registered [ 3.170944] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.174313] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.178034] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.181181] ACPI: Power Button [PWRF] [ 3.188542] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.196185] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.213426] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.223305] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.241289] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.269088] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.299742] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.306745] Non-volatile memory driver v1.3 [ 3.308415] Linux agpgart interface v0.103 [ 3.343127] virtio_blk virtio1: [vda] 145792 512-byte logical blocks (74.6 MB/71.2 MiB) [ 3.346944] vda: detected capacity change from 0 to 74645504 [ 3.361193] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.364272] vdb: detected capacity change from 0 to 1073741824 [ 3.378736] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.382074] vdc: detected capacity change from 0 to 2621440000 [ 3.397635] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.400416] vdd: detected capacity change from 0 to 2621440000 [ 3.414976] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.418581] vde: detected capacity change from 0 to 4294967296 [ 3.433076] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.436347] vdf: detected capacity change from 0 to 4294967296 [ 3.442889] libphy: Fixed MDIO Bus: probed [ 3.448410] usbcore: registered new interface driver usbserial_generic [ 3.451728] usbserial: USB Serial support registered for generic [ 3.454521] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.459484] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.461700] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.464664] mousedev: PS/2 mouse device common for all mice [ 3.467516] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.470472] rtc_cmos 00:05: RTC can wake from S4 [ 3.475477] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.477458] rtc_cmos 00:05: registered as rtc0 [ 3.481939] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.485507] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.485548] intel_pstate: CPU model not supported [ 3.487487] hid: raw HID events driver (C) Jiri Kosina [ 3.500419] usbcore: registered new interface driver usbhid [ 3.502642] usbhid: USB HID core driver [ 3.504647] drop_monitor: Initializing network drop monitor service [ 3.507575] Initializing XFRM netlink socket [ 3.509760] NET: Registered protocol family 10 [ 3.513275] Segment Routing with IPv6 [ 3.515027] NET: Registered protocol family 17 [ 3.516966] mpls_gso: MPLS GSO support [ 3.523901] RAS: Correctable Errors collector initialized. [ 3.526753] AVX version of gcm_enc/dec engaged. [ 3.528906] AES CTR mode by8 optimization enabled [ 3.595707] sched_clock: Marking stable (3595649291, 0)->(4569942324, -974293033) [ 3.599192] registered taskstats version 1 [ 3.601337] Loading compiled-in X.509 certificates [ 3.603573] zswap: loaded using pool lzo/zbud [ 3.631119] Key type big_key registered [ 3.644846] Key type encrypted registered [ 3.646653] ima: No TPM chip found, activating TPM-bypass! [ 3.648710] ima: Allocated hash algorithm: sha1 [ 3.650707] ima: No architecture policies found [ 3.652756] evm: Initialising EVM extended attributes: [ 3.654943] evm: security.selinux [ 3.656294] evm: security.ima [ 3.657681] evm: security.capability [ 3.659154] evm: HMAC attrs: 0x1 [ 3.661718] rtc_cmos 00:05: setting system clock to 2026-07-16 06:54:15 UTC (1784184855) [ 3.667898] debug: unmapping init [mem 0xffffffffbf003000-0xffffffffbf1fffff] [ 3.671428] debug: unmapping init [mem 0xffffffffbdd82000-0xffffffffbe058fff] [ 3.687734] Write protecting the kernel read-only data: 28672k [ 3.690786] debug: unmapping init [mem 0xffffffffbc403000-0xffffffffbc5fffff] [ 3.694344] debug: unmapping init [mem 0xffffffffbcd14000-0xffffffffbcdfffff] [ 3.728859] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.737921] systemd[1]: Detected virtualization kvm. [ 3.740013] systemd[1]: Detected architecture x86-64. [ 3.742161] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.769339] systemd[1]: No hostname configured. [ 3.771256] systemd[1]: Set hostname to . [ 3.773406] random: systemd: uninitialized urandom read (16 bytes read) [ 3.776056] systemd[1]: Initializing machine ID from random generator. [ 3.809049] random: ln: uninitialized urandom read (6 bytes read) [ 3.905848] random: systemd: uninitialized urandom read (16 bytes read) [ 3.908979] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.913283] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.918391] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Sockets. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Reached target Timers. Starting Setup Virtual Console... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.532065] device-mapper: uevent: version 1.0.3 [ 4.534225] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] [ 5.157656] random: fast init done Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.362582] virtio_net virtio0 ens2: renamed from eth0 [ 5.435643] scsi host0: ata_piix [ 5.507545] scsi host1: ata_piix [ 5.511270] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.514188] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.283758] dracut-initqueue[584]: RTNETLINK answers: File exists [ 10.164356] random: crng init done [ 10.165899] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.752838] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.036666] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.318525] SELinux: Disabled at runtime. [ 12.387559] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.395950] systemd[1]: Detected virtualization kvm. [ 12.397939] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.963266] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.967095] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.976829] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.982655] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.986961] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.000674] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.012289] systemd[1]: Mounting Kernel Debug File System... Mounting Kernel Debug File System... Activating swap /dev/disk/by-label/SWAP... Mounting Huge Pages File System... [ 13.057276] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting POSIX Message Queue File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Paths. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-sshd\x2dkeygen.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target rpc_pipefs.target. Starting Remount Root and Kernel File Systems... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.518317] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.800771] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.888650] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.110567] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.123569] EDAC sbridge: Ver: 1.1.2 [ 16.150097] Key type dns_resolver registered [ 16.467573] NFS: Registering the id_resolver key type [ 16.469336] Key type id_resolver registered [ 16.472439] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... Starting Network Manager... [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg446-server login: [ 44.323112] libcfs: loading out-of-tree module taints kernel. [ 44.345403] Key type ._llcrypt registered [ 44.347339] Key type .llcrypt registered [ 44.408442] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 54.019244] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 54.928963] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 54.963090] alg: No test for adler32 (adler32-zlib) [ 56.766826] Lustre: Lustre: Build Version: 2.17.54_162_gdc75e25 [ 57.873432] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 59.712359] Key type lgssc registered [ 62.515903] Lustre: Echo OBD driver; http://www.lustre.org/ [ 86.579606] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 91.540139] hrtimer: interrupt took 3168823 ns [ 129.709012] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 146.165883] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 146.179563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 147.687680] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 147.728233] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 147.868140] Lustre: lustre-MDT0000: new disk, initializing [ 147.985038] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 148.048566] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 153.117736] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 168.761789] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 168.943747] Lustre: 6487:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 169.000535] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 169.020179] Lustre: Skipped 1 previous similar message [ 169.173454] Lustre: lustre-MDT0001: new disk, initializing [ 169.298754] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 169.354906] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 169.371431] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 174.969188] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 180.181599] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 190.550891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 190.878543] Lustre: lustre-OST0000: new disk, initializing [ 190.882818] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 190.887783] Lustre: 8423:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 190.976055] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 191.956108] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 191.975850] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 192.064750] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 197.520691] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 213.749170] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 213.889912] Lustre: lustre-OST0001: new disk, initializing [ 213.893904] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 213.901213] Lustre: 9495:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 214.002176] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 220.744283] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 222.754300] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 222.765620] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 222.808115] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 233.104711] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 239.046745] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 245.788703] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing check_logdir /tmp/testlogs/ [ 251.615396] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing yml_node [ 256.927494] Lustre: DEBUG MARKER: Client: 2.17.54.162 [ 259.556609] Lustre: DEBUG MARKER: MDS: 2.17.54.162 [ 262.214288] Lustre: DEBUG MARKER: OSS: 2.17.54.162 [ 264.094973] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Jul 16 02:58:33 EDT 2026 [ 283.783885] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 285.678576] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 302.561427] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 302.569581] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 302.588704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 304.611657] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 304.612906] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 304.633512] Lustre: Skipped 3 previous similar messages [ 307.796309] Lustre: server umount lustre-MDT0000 complete [ 311.314625] LustreError: 9494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784185163 with bad export cookie 9499392247995198251 [ 311.318972] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 311.323384] LustreError: 9494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 314.851935] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 314.855585] LustreError: 8960:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 314.869173] Lustre: Skipped 2 previous similar messages [ 314.874154] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 314.891577] LustreError: 8960:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 314.910417] Lustre: Skipped 1 previous similar message [ 316.908394] LustreError: 8960:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 316.908676] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 316.930651] LustreError: 8960:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 316.938508] Lustre: Skipped 1 previous similar message [ 317.761849] Lustre: server umount lustre-MDT0001 complete [ 322.196499] Lustre: server umount lustre-OST0000 complete [ 326.424745] Lustre: server umount lustre-OST0001 complete [ 334.076413] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 347.013446] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 388.188985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 388.482457] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 388.523770] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 388.581412] Lustre: lustre-MDT0000: new disk, initializing [ 388.648957] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 388.687151] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 393.297537] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 404.431948] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 404.567306] Lustre: 15057:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 404.629403] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 404.633761] Lustre: Skipped 1 previous similar message [ 404.761476] Lustre: lustre-MDT0001: new disk, initializing [ 404.855240] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 404.910165] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 404.925347] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 410.035144] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 420.450567] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 426.909753] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 435.163188] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 435.675449] Lustre: lustre-OST0000: new disk, initializing [ 435.687379] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 435.696095] Lustre: 17039:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 435.832645] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 437.346725] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 437.361627] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 437.479879] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 442.552762] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 454.219392] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 459.927554] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 460.236670] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 466.613398] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 467.001252] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 473.574679] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 473.576128] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 473.581589] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 473.585556] LustreError: Skipped 1 previous similar message [ 473.634490] Lustre: Skipped 2 previous similar messages [ 475.726339] Lustre: server umount lustre-OST0000 complete [ 481.766250] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 481.775621] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 481.793111] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 481.803933] Lustre: Skipped 1 previous similar message [ 486.522940] Lustre: server umount lustre-MDT0000 complete [ 488.930819] LustreError: 15067:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 488.958310] LustreError: 15067:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 491.039569] LustreError: 15047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784185342 with bad export cookie 9499392247995199784 [ 491.042266] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 491.052316] LustreError: 15047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 491.550898] Lustre: server umount lustre-MDT0001 complete [ 498.525973] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 03:02:28 (1784185348) [ 521.009864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 521.675813] LustreError: 19915:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 521.804707] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 526.831502] LustreError: 19916:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 527.424634] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 537.057868] LustreError: 19915:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 537.083033] LustreError: 19915:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 538.633130] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 544.749842] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 554.026306] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 561.878653] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 564.765831] Lustre: Failing over lustre-MDT0000 [ 565.163700] Lustre: server umount lustre-MDT0000 complete [ 568.804281] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 568.808520] LustreError: 20855:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 568.830311] Lustre: Skipped 2 previous similar messages [ 585.056809] Lustre: 3620:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784185420/real 1784185420] req@ffff93cfb7908a80 x1870853475301632/t0(0) o400->MGC192.168.204.146@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1784185436 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 585.101558] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 585.962952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 594.890710] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 594.914952] Lustre: Skipped 1 previous similar message [ 600.038766] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 600.049214] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 600.070782] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 600.188183] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 611.571146] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 618.471884] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 618.480873] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 618.488943] Lustre: Skipped 1 previous similar message [ 618.505743] Lustre: Skipped 3 previous similar messages [ 622.024684] Lustre: server umount lustre-MDT0000 complete [ 623.595550] LustreError: 20856:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 623.634526] LustreError: 20856:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 627.433907] LustreError: 19897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784185479 with bad export cookie 9499392247995201198 [ 627.445757] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 627.448639] LustreError: 19897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 628.046918] Lustre: server umount lustre-MDT0001 complete [ 640.366189] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 647.654773] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 658.605212] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 665.500828] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 674.635947] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 681.995732] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 685.015307] Lustre: Failing over lustre-MDT0000 [ 685.026286] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -19 [ 685.038310] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 685.055801] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 685.740460] Lustre: server umount lustre-MDT0000 complete [ 689.123127] LustreError: 23986:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 689.148657] LustreError: 23986:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 704.480139] Lustre: 3622:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784185540/real 1784185540] req@ffff93cfb8d04700 x1870853475333632/t0(0) o400->MGC192.168.204.146@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1784185556 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 704.532120] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 709.535128] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 714.722741] LustreError: 3618:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff93cfb7942300 x1870853475336320/t0(0) o250->MGC192.168.204.146@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 715.356614] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 720.359441] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 720.399213] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 720.408678] Lustre: Skipped 1 previous similar message [ 720.463038] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 722.326251] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 735.184739] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 741.347683] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 746.737177] Lustre: server umount lustre-MDT0000 complete [ 751.637956] LustreError: 23238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784185603 with bad export cookie 9499392247995202136 [ 751.650582] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 751.652703] LustreError: 23238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 752.593724] Lustre: server umount lustre-MDT0001 complete [ 764.936183] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 03:06:54 (1784185614) [ 790.058614] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 803.702573] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 851.896728] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 864.052417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 864.263747] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 864.292105] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 864.354313] Lustre: lustre-MDT0000: new disk, initializing [ 864.415275] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 864.420524] Lustre: Skipped 3 previous similar messages [ 864.432231] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 869.214391] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 881.421935] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 881.540553] Lustre: 29638:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 881.957381] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 887.332131] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 899.993505] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 907.050928] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 916.391803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 916.776389] Lustre: 31609:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 918.257049] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 918.314957] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 918.402879] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 918.420246] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 924.610070] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 936.285953] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 942.917451] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 969.699269] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 969.711541] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 969.715820] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 969.715832] Lustre: Skipped 3 previous similar messages [ 969.720953] Lustre: Skipped 4 previous similar messages [ 975.430728] Lustre: server umount lustre-OST0000 complete [ 994.272095] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 994.543575] Lustre: server umount lustre-MDT0000 complete [ 995.302837] LustreError: 29644:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 995.326926] LustreError: 29644:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 15 previous similar messages [ 999.871740] LustreError: 29630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784185851 with bad export cookie 9499392247995202542 [ 999.882326] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 999.894706] LustreError: 29630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 1000.469678] Lustre: server umount lustre-MDT0001 complete [ 1014.155694] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 1018.127689] Key type lgssc unregistered [ 1018.568787] LNet: 33594:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1018.589236] LNetError: 33594:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1018.610913] LNet: Removed LNI 192.168.204.146@tcp [ 1019.743213] Key type .llcrypt unregistered [ 1019.748556] Key type ._llcrypt unregistered [ 1047.912729] Key type ._llcrypt registered [ 1047.918119] Key type .llcrypt registered [ 1048.004073] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 1049.019852] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1049.066418] alg: No test for adler32 (adler32-zlib) [ 1050.245654] Lustre: Lustre: Build Version: 2.17.54_162_gdc75e25 [ 1050.544994] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 1052.272497] Key type lgssc registered [ 1053.853547] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1065.506614] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1065.523435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1067.289650] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1073.306249] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1077.739529] LustreError: 35010:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1077.795912] LustreError: 35010:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1082.850781] LustreError: 35011:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1084.851261] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1085.464114] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1091.502241] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1100.798834] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1107.355632] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1117.813974] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1118.463597] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1129.334717] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1130.838202] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1138.062338] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1140.540283] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1156.072455] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 03:13:25 (1784186005) [ 1159.087376] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1161.481527] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1169.784271] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 03:13:39 (1784186019) [ 1181.861619] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 03:13:50 (1784186030) [ 1183.173191] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1194.240202] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1194.248528] Lustre: Skipped 2 previous similar messages [ 1196.521726] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 03:14:06 (1784186046) [ 1198.994168] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1199.003912] Lustre: Skipped 2 previous similar messages [ 1208.453404] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1208.457022] Lustre: Skipped 4 previous similar messages [ 1225.384663] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 03:14:35 (1784186075) [ 1228.477571] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1232.730599] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1242.862166] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 03:14:52 (1784186092) [ 1245.232145] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1245.236332] Lustre: Skipped 6 previous similar messages [ 1260.407192] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 03:15:09 (1784186109) [ 1279.466216] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1279.477447] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1279.497446] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1281.516695] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1281.517864] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1281.527430] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1284.329369] Lustre: server umount lustre-OST0000 complete [ 1290.212252] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1290.238753] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1290.270595] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1290.284174] Lustre: Skipped 1 previous similar message [ 1296.004939] Lustre: server umount lustre-MDT0000 complete [ 1296.876231] LustreError: 35011:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1296.907941] LustreError: 35011:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1300.505802] LustreError: 34992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784186152 with bad export cookie 1311832118706508426 [ 1300.507304] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1300.523498] LustreError: 34992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1300.897036] Lustre: server umount lustre-MDT0001 complete [ 1313.360779] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 1317.030280] Key type lgssc unregistered [ 1317.493849] LNet: 42873:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1317.515710] LNetError: 42873:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1317.535763] LNet: Removed LNI 192.168.204.146@tcp [ 1318.576336] Key type .llcrypt unregistered [ 1318.577938] Key type ._llcrypt unregistered [ 1339.122438] Key type ._llcrypt registered [ 1339.124718] Key type .llcrypt registered [ 1339.254615] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 1356.054625] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 1357.689038] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1357.830495] alg: No test for adler32 (adler32-zlib) [ 1359.113937] Lustre: Lustre: Build Version: 2.17.54_162_gdc75e25 [ 1359.464054] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 1361.192227] Key type lgssc registered [ 1362.853738] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1407.749045] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1407.841215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1409.305879] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1409.373493] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1409.569355] Lustre: lustre-MDT0000: new disk, initializing [ 1409.749665] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1409.775631] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1415.639827] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1427.314844] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1427.431368] Lustre: 46029:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1427.485561] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1427.489255] Lustre: Skipped 1 previous similar message [ 1427.558727] Lustre: lustre-MDT0001: new disk, initializing [ 1427.666033] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1427.738130] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1427.752804] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1432.982209] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1444.129264] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1450.996848] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1458.810271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1459.090547] Lustre: lustre-OST0000: new disk, initializing [ 1459.099723] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1459.110565] Lustre: 48010:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1459.224045] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1465.933195] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1465.947511] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1466.101386] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1466.800746] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1478.157784] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1483.455895] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1483.661983] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1489.261190] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1489.491578] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1497.064210] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1497.067742] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1497.073582] LustreError: Skipped 1 previous similar message [ 1497.081147] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1497.095945] Lustre: Skipped 1 previous similar message [ 1497.128629] Lustre: Skipped 1 previous similar message [ 1498.569556] Lustre: server umount lustre-OST0000 complete [ 1504.741632] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1504.761468] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1504.783937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1507.297852] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1507.299569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1507.330423] Lustre: Skipped 1 previous similar message [ 1509.726381] Lustre: server umount lustre-MDT0000 complete [ 1514.366985] LustreError: 46065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784186366 with bad export cookie 1557183946127591877 [ 1514.381797] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1514.386402] LustreError: 46065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1514.809568] Lustre: server umount lustre-MDT0001 complete [ 1525.944871] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 03:19:35 (1784186375) [ 1533.348968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1534.003942] LustreError: 50470:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1534.068487] LustreError: 50470:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1534.222414] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1539.524418] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1539.557331] LustreError: 50471:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1543.656270] LustreError: 50470:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1548.773174] LustreError: 50471:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1549.583178] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1550.262758] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1555.975748] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1565.995968] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1573.448962] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1581.630675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1581.983604] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1589.216683] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1597.712863] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1606.768728] LustreError: 53002:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1607.702651] Lustre: Modifying parameter general.timeout=1 in log params [ 1608.772955] Lustre: Modifying parameter general.timeout=2 in log params [ 1611.037850] Lustre: Modifying parameter general.timeout=4 in log params [ 1611.043784] Lustre: Skipped 1 previous similar message [ 1615.318653] Lustre: Modifying parameter general.timeout=8 in log params [ 1615.326678] Lustre: Skipped 3 previous similar messages [ 1624.053315] Lustre: Modifying parameter general.timeout=16 in log params [ 1624.058114] Lustre: Skipped 7 previous similar messages [ 1640.120944] Lustre: Modifying parameter general.timeout=33 in log params [ 1640.127503] Lustre: Skipped 16 previous similar messages [ 1672.693278] Lustre: Modifying parameter general.timeout=68 in log params [ 1672.702994] Lustre: Skipped 34 previous similar messages [ 1740.861207] Lustre: Disabling parameter general.timeout= in log params [ 1740.867741] Lustre: Skipped 61 previous similar messages [ 1743.978963] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 03:23:13 (1784186593) [ 1747.375963] Lustre: 59953:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1749.399479] Lustre: 60062:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1752.965336] Lustre: 60259:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1752.974288] Lustre: 60259:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1789.605034] Lustre: 61835:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1789.614774] Lustre: 61835:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1789.636281] LustreError: 61835:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1793.458464] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 03:24:03 (1784186643) [ 1809.382314] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1809.395508] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1809.404811] Lustre: Skipped 2 previous similar messages [ 1809.408530] Lustre: Skipped 1 previous similar message [ 1813.563204] Lustre: server umount lustre-MDT0000 complete [ 1814.499202] LustreError: 50466:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1814.528835] LustreError: 50466:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1817.569362] LustreError: 53349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784186669 with bad export cookie 1557183946127592759 [ 1817.573543] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1817.581223] LustreError: 53349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1818.013828] Lustre: server umount lustre-MDT0001 complete [ 1832.336692] Lustre: server umount lustre-OST0000 complete [ 1864.571381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1865.067662] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1865.340780] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1870.002509] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1878.966196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1879.027166] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1879.052554] Lustre: 64367:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1883.873192] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1892.800698] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1898.750558] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1906.323051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1906.510494] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1906.787668] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1906.795809] Lustre: Skipped 1 previous similar message [ 1913.640514] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1921.845281] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1924.019895] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1924.026440] Lustre: Skipped 23 previous similar messages [ 1924.080453] Lustre: 66933:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1935.841462] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1935.857715] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1935.873214] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1938.147508] Lustre: server umount lustre-OST0000 complete [ 1946.087847] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1946.107598] LustreError: Skipped 1 previous similar message [ 1946.116213] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1946.136355] Lustre: Skipped 1 previous similar message [ 1946.150485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1946.157680] Lustre: Skipped 2 previous similar messages [ 1948.753787] Lustre: server umount lustre-MDT0000 complete [ 1951.719394] LustreError: 64373:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1951.752855] LustreError: 64373:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1952.783318] LustreError: 64359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784186804 with bad export cookie 1557183946127595958 [ 1952.787328] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1952.799340] LustreError: 64359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1953.198191] Lustre: server umount lustre-MDT0001 complete [ 1965.642517] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 1968.301578] Key type lgssc unregistered [ 1968.586730] LNet: 68146:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1968.597166] LNetError: 68146:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1968.615949] LNet: Removed LNI 192.168.204.146@tcp [ 1969.395930] Key type .llcrypt unregistered [ 1969.400916] Key type ._llcrypt unregistered [ 1992.329310] Key type ._llcrypt registered [ 1992.331124] Key type .llcrypt registered [ 1992.450438] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 1993.709912] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1993.749839] alg: No test for adler32 (adler32-zlib) [ 1994.876444] Lustre: Lustre: Build Version: 2.17.54_162_gdc75e25 [ 1995.197047] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 1996.930936] Key type lgssc registered [ 1998.285569] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2007.254647] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2007.271281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2008.922702] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2013.508457] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2018.277173] LustreError: 69561:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2018.310214] LustreError: 69561:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2022.517715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2023.113297] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2026.927850] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2034.839439] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2040.734931] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2047.713937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2047.908219] LustreError: 71387:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2048.250720] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2054.781204] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2063.537386] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2071.087967] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2084.321692] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2084.326742] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2084.329921] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2084.355456] Lustre: Skipped 2 previous similar messages [ 2087.032644] Lustre: server umount lustre-MDT0000 complete [ 2089.450731] LustreError: 69561:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2089.477758] LustreError: 69561:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2090.878916] LustreError: 69543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784186942 with bad export cookie 3057987695350477509 [ 2090.880634] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2090.887091] LustreError: 69543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2091.173189] Lustre: server umount lustre-MDT0001 complete [ 2105.743938] Lustre: server umount lustre-OST0000 complete [ 2136.799720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2137.195773] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2137.244781] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2137.582050] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2142.356650] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2151.989237] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2152.188230] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2152.235612] Lustre: 74435:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2152.888017] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2157.836087] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2166.709704] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2173.127900] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2179.740536] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2179.906123] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2180.189132] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2185.441785] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2192.113978] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2193.727400] Lustre: 77010:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2195.510909] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2200.033203] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2200.039375] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2200.050645] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2200.059958] Lustre: Skipped 2 previous similar messages [ 2205.676722] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2205.679535] Lustre: Skipped 1 previous similar message [ 2206.168661] Lustre: server umount lustre-OST0000 complete [ 2210.785414] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2210.808183] Lustre: Skipped 1 previous similar message [ 2210.820489] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2210.827134] Lustre: Skipped 1 previous similar message [ 2215.909509] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2215.919351] Lustre: Skipped 1 previous similar message [ 2216.624517] Lustre: server umount lustre-MDT0000 complete [ 2220.453304] LustreError: 75180:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784187072 with bad export cookie 3057987695350478349 [ 2220.454716] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2220.464637] LustreError: 75180:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2220.849630] Lustre: server umount lustre-MDT0001 complete [ 2231.310987] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 2234.106236] Key type lgssc unregistered [ 2234.439172] LNet: 78217:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2234.449596] LNetError: 78217:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2234.479906] LNet: Removed LNI 192.168.204.146@tcp [ 2235.417919] Key type .llcrypt unregistered [ 2235.424564] Key type ._llcrypt unregistered [ 2261.552258] Key type ._llcrypt registered [ 2261.554709] Key type .llcrypt registered [ 2261.667668] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 2262.854373] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2262.904644] alg: No test for adler32 (adler32-zlib) [ 2263.954451] Lustre: Lustre: Build Version: 2.17.54_162_gdc75e25 [ 2264.215354] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 2265.904321] Key type lgssc registered [ 2267.224400] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2275.489380] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2275.508659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2276.913707] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2281.252676] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2287.083974] LustreError: 79631:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2287.108885] LustreError: 79631:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2290.430093] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2290.999024] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2295.444631] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2302.999930] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2308.792876] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2315.302675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2315.889640] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2321.985399] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2329.552560] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2336.385840] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2337.217868] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2339.041108] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2339.050570] Lustre: Skipped 1 previous similar message [ 2341.508955] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2341.517273] Lustre: Skipped 2 previous similar messages [ 2345.661247] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2345.666232] Lustre: Skipped 4 previous similar messages [ 2353.939791] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2353.943958] Lustre: Skipped 9 previous similar messages [ 2370.216762] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2370.220897] Lustre: Skipped 17 previous similar messages [ 2403.075940] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2403.085743] Lustre: Skipped 32 previous similar messages [ 2439.934066] Lustre: 87733:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2443.023759] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 03:34:52 (1784187292) [ 2444.665868] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2446.454149] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 03:34:56 (1784187296) [ 2454.497900] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2454.509811] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2454.541104] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2455.529588] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2455.532048] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2455.556046] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2455.571301] Lustre: Skipped 1 previous similar message [ 2460.646177] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2460.652278] Lustre: Skipped 1 previous similar message [ 2465.248790] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2465.456146] Lustre: server umount lustre-OST0000 complete [ 2469.860264] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2469.867646] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2469.893887] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2475.776596] Lustre: server umount lustre-MDT0000 complete [ 2476.007023] LustreError: 79632:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2476.027915] LustreError: 79632:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2479.668365] LustreError: 79612:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784187331 with bad export cookie 2399769672292688278 [ 2479.672581] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2479.678629] LustreError: 79612:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2480.013727] Lustre: server umount lustre-MDT0001 complete [ 2491.777944] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 2494.905943] Key type lgssc unregistered [ 2495.219463] LNet: 88983:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2495.233336] LNetError: 88983:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2496.294678] LNet: Removed LNI 192.168.204.146@tcp [ 2497.170583] Key type .llcrypt unregistered [ 2497.173160] Key type ._llcrypt unregistered [ 2501.947220] Key type ._llcrypt registered [ 2501.953559] Key type .llcrypt registered [ 2502.068153] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2506.773852] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules [ 2507.839100] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2507.888088] alg: No test for adler32 (adler32-zlib) [ 2509.006746] Lustre: Lustre: Build Version: 2.17.54_162_gdc75e25 [ 2509.238718] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 2510.904159] Key type lgssc registered [ 2512.702809] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2513.816044] LustreError: 90186:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 2532.371898] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 2535.730066] LustreError: 90186:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2541.046920] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2541.067157] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2542.642743] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2547.768457] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2552.802974] LustreError: 91208:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2552.844979] LustreError: 91208:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2557.931426] LustreError: 91209:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2558.386655] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 03:36:48 (1784187408) [ 2563.049277] LustreError: 91208:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2564.055048] LustreError: 91240:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2564.077940] LustreError: 91240:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2564.087630] LustreError: 91240:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 21, retries 0, failed: rc = -5 [ 2564.464555] Lustre: server umount lustre-MDT0000 complete [ 2577.737285] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 2580.431423] Key type lgssc unregistered [ 2580.763722] LNet: 92735:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2580.770694] LNetError: 92735:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2580.782698] LNet: Removed LNI 192.168.204.146@tcp [ 2581.524459] Key type .llcrypt unregistered [ 2581.526754] Key type ._llcrypt unregistered [ 2601.246328] Key type ._llcrypt registered [ 2601.251246] Key type .llcrypt registered [ 2601.370829] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 2602.173576] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2602.333708] alg: No test for adler32 (adler32-zlib) [ 2603.374791] Lustre: Lustre: Build Version: 2.17.54_162_gdc75e25 [ 2603.650091] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 2605.330659] Key type lgssc registered [ 2606.734878] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2616.597479] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2616.615842] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2618.252638] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2623.333347] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2627.555717] LustreError: 93956:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2627.586047] LustreError: 93956:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2632.675195] LustreError: 93955:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2633.701300] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2634.440391] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2639.246743] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2646.949966] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2653.083441] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2660.177493] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2660.773721] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2666.911829] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2674.451454] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2727.949849] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 03:39:37 (1784187577) [ 2729.885367] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2731.856241] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 03:39:41 (1784187581) [ 2736.609559] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2736.623579] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2736.641970] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2739.686873] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2739.698110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2739.710611] Lustre: Skipped 1 previous similar message [ 2739.740781] Lustre: Skipped 2 previous similar messages [ 2742.396350] Lustre: server umount lustre-MDT0000 complete [ 2744.806231] LustreError: 93950:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2744.836944] LustreError: 93950:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2746.675742] LustreError: 93935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784187598 with bad export cookie 1383675056605185753 [ 2746.684200] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2746.702545] LustreError: 93935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2747.099763] Lustre: server umount lustre-MDT0001 complete [ 2760.556426] Lustre: server umount lustre-OST0000 complete [ 2772.537929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2773.016826] LustreError: 98251:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2773.223352] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2777.795094] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2778.594433] LustreError: 98252:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2786.551104] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2787.202750] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2791.638828] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2800.093627] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2806.641549] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2820.865043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2821.129183] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2821.135507] LustreError: 98240:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2821.152601] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2821.160604] LustreError: 100336:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2821.164850] LustreError: 100336:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2821.171737] LustreError: 100336:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2821.257642] Lustre: server umount lustre-OST0000 complete [ 2821.268366] LustreError: 100336:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2830.502559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2830.762087] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2830.768737] LustreError: 98238:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2830.807913] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2830.826465] LustreError: 100802:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2830.832635] LustreError: 100802:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2830.839872] LustreError: 100802:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2830.934127] Lustre: server umount lustre-OST0000 complete [ 2830.936883] LustreError: 100802:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2841.413850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2841.610319] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2841.613726] Lustre: Found index 0 for lustre-OST0000, updating log [ 2841.624812] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2841.712956] Lustre: lustre-OST0000: new disk, initializing [ 2841.719352] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2841.735434] Lustre: 101340:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2842.045888] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2848.973300] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2853.662151] Lustre: Failing over lustre-OST0000 [ 2853.821586] Lustre: server umount lustre-OST0000 complete [ 2858.465592] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2858.472174] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2858.492963] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2862.059428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2862.068409] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2867.170259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2867.179462] Lustre: Skipped 2 previous similar messages [ 2868.309753] Lustre: server umount lustre-MDT0000 complete [ 2872.244400] LustreError: 98232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784187724 with bad export cookie 1383675056605188504 [ 2872.253061] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2872.259961] LustreError: 98232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2872.767537] Lustre: server umount lustre-MDT0001 complete [ 2882.866323] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 03:42:12 (1784187732) [ 2904.032795] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 2915.905993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2916.261865] LustreError: 103699:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2916.277747] LustreError: 103699:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2916.351916] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2921.689431] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2932.468090] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2938.007661] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2941.181576] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2948.684383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2949.313044] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2949.318891] Lustre: Skipped 1 previous similar message [ 2949.331859] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2956.240725] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2957.483635] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2957.495595] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 2957.518434] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2961.644109] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:05 [ 2961.677387] Lustre: lustre-OST0000: Recovery over after 0:04, of 1 clients 1 recovered and 0 were evicted. [ 2961.695898] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2961.712387] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2961.713284] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2965.155227] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2965.261603] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2965.380819] Lustre: lustre-OST0001: new disk, initializing [ 2965.385372] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2965.393754] Lustre: 106021:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2972.362058] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2973.269858] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2973.285539] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2973.386408] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2980.240345] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2985.319814] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2990.050871] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2990.067332] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2990.099153] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3004.384395] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3004.674396] Lustre: server umount lustre-MDT0001 complete [ 3005.599444] LustreError: 106032:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3005.628837] LustreError: 106032:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3015.827554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3015.951329] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3015.956741] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3016.016847] Lustre: 103688:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3016.828377] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3016.839606] Lustre: Skipped 1 previous similar message [ 3017.321745] LustreError: 107558:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3017.335624] LustreError: 107558:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3017.344052] LustreError: 103691:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3017.441764] LustreError: 107566:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.146@tcp: cfg command failed: rc = -17 [ 3017.449181] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3017.449181] [ 3021.419822] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3022.311625] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3031.162589] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 03:44:41 (1784187881) [ 3050.254880] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 3068.349296] Lustre: 109637:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3091.997877] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3095.274965] Lustre: 110771:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3190.241274] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3190.252429] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3190.266529] Lustre: Skipped 2 previous similar messages [ 3190.276243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3190.288357] Lustre: Skipped 12 previous similar messages [ 3193.449249] Lustre: server umount lustre-MDT0000 complete [ 3196.392562] LustreError: 103699:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3196.430390] LustreError: 103699:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 3197.952300] LustreError: 106789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784188049 with bad export cookie 1383675056605189505 [ 3197.964794] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3197.967953] LustreError: 106789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3198.411909] Lustre: server umount lustre-MDT0001 complete [ 3213.093966] Lustre: server umount lustre-OST0000 complete [ 3217.392674] Lustre: 93266:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784188053/real 1784188053] req@ffff93ce88781500 x1870856145187328/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784188069 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3217.429836] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3217.456924] Lustre: Skipped 3 previous similar messages [ 3217.831974] Lustre: server umount lustre-OST0001 complete [ 3223.822806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3234.731796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3243.891967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3254.786706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3275.725383] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 3286.377737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3286.405121] Lustre: lustre-MDT0000: reset Object Index mappings [ 3286.892407] LustreError: 114725:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3287.044920] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3292.137669] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3302.086898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3302.138498] Lustre: lustre-MDT0001: reset Object Index mappings [ 3308.815888] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3312.415365] Lustre: 115875:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3321.828872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3327.941266] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 3329.663659] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3332.055416] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:387 to 0x280000401:418) [ 3339.740467] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3345.938918] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 3345.949945] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 3348.028311] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3356.636470] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3361.452268] Lustre: 117750:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3421.083919] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 03:51:11 (1784188271) [ 3448.292374] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3448.311164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3448.316367] Lustre: Skipped 2 previous similar messages [ 3448.327829] Lustre: Skipped 7 previous similar messages [ 3454.053802] Lustre: server umount lustre-MDT0000 complete [ 3458.033255] LustreError: 115878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784188309 with bad export cookie 1383675056605236601 [ 3458.035641] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3458.050618] LustreError: 115878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3458.539085] LustreError: 117898:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3458.562555] LustreError: 117898:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 3464.670319] Lustre: server umount lustre-MDT0001 complete [ 3475.702092] Lustre: server umount lustre-OST0000 complete [ 3486.079383] Lustre: server umount lustre-OST0001 complete [ 3493.732392] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 3505.908716] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 3548.616496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3548.891050] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3548.940699] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3549.040456] Lustre: lustre-MDT0000: new disk, initializing [ 3549.109384] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3549.113033] Lustre: Skipped 3 previous similar messages [ 3549.129061] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3554.376326] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3562.765127] Lustre: server umount lustre-MDT0000 complete [ 3574.611080] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3575.151744] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3575.169750] Lustre: 122026:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3575.188298] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3581.145699] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3586.666202] Lustre: server umount lustre-MDT0000 complete [ 3597.024269] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 03:54:06 (1784188446) [ 3599.264859] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3601.280785] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 03:54:11 (1784188451) [ 3622.243245] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 3634.082768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3640.495418] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3650.347142] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3650.550989] Lustre: 124078:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3650.613311] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3650.617830] Lustre: Skipped 1 previous similar message [ 3650.685805] Lustre: lustre-MDT0001: new disk, initializing [ 3650.819816] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3650.841667] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3656.506638] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3662.187489] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3670.568874] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3671.073920] Lustre: lustre-OST0000: new disk, initializing [ 3671.080617] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3671.093315] Lustre: 125654:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3672.581887] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3672.612261] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3672.831051] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3679.685182] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3692.922936] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3693.178635] Lustre: lustre-OST0001: new disk, initializing [ 3693.188485] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3693.199220] Lustre: 126525:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3695.257717] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3695.273175] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3695.475925] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3700.516626] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3712.095070] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3717.376703] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4178.782210] Lustre: 132969:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4178.790812] Lustre: 132969:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4178.799383] Lustre: 132969:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4178.806885] Lustre: 132969:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4178.814868] Lustre: 132969:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4178.823333] Lustre: 132969:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4517.045722] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 04:09:26 (1784189366) [ 4535.276314] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4535.290758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4535.298034] Lustre: Skipped 5 previous similar messages [ 4535.311989] Lustre: Skipped 11 previous similar messages [ 4536.099554] Lustre: server umount lustre-MDT0000 complete [ 4540.386170] LustreError: 127332:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4540.409195] LustreError: 127332:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 4540.902715] LustreError: 124070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784189392 with bad export cookie 1383675056605307098 [ 4540.905715] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4540.914565] LustreError: 124070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4545.506564] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4545.518711] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4545.543076] Lustre: Skipped 2 previous similar messages [ 4545.561170] Lustre: Skipped 1 previous similar message [ 4547.996381] Lustre: server umount lustre-MDT0001 complete [ 4559.506142] Lustre: server umount lustre-OST0000 complete [ 4578.272822] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4578.620997] Lustre: server umount lustre-OST0001 complete [ 4586.724781] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 4598.486510] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 4639.108726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4639.442168] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4639.538972] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4639.735866] Lustre: lustre-MDT0000: new disk, initializing [ 4639.915861] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4639.929403] Lustre: Skipped 5 previous similar messages [ 4639.980655] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4646.022172] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4659.792442] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4659.972944] Lustre: 136033:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4660.044586] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4660.048351] Lustre: Skipped 1 previous similar message [ 4660.154504] Lustre: lustre-MDT0001: new disk, initializing [ 4660.343714] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4660.350624] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4666.508224] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4678.847476] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4686.217298] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4695.250162] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4695.648464] Lustre: lustre-OST0000: new disk, initializing [ 4695.653642] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4695.672648] Lustre: 138008:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4697.149909] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4697.174646] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4697.322175] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4702.755948] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4714.095457] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4726.700855] Lustre: lustre-MDD0000: changelog on [ 4730.149187] Lustre: lustre-MDD0001: changelog on [ 7037.579491] Lustre: lustre-MDD0001: changelog off [ 7039.901067] Lustre: 136041:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 7040.940791] Lustre: lustre-MDD0000: changelog off [ 7052.780172] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7052.794277] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7052.810982] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7052.819660] Lustre: Skipped 1 previous similar message [ 7054.892770] Lustre: server umount lustre-OST0000 complete [ 7061.473991] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7061.486283] LustreError: Skipped 1 previous similar message [ 7061.493172] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7061.521541] Lustre: Skipped 1 previous similar message [ 7061.527380] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7065.942350] Lustre: server umount lustre-MDT0000 complete [ 7068.132184] LustreError: 136047:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7068.168912] LustreError: 136047:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7070.304624] LustreError: 136028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784191922 with bad export cookie 1383675056606127148 [ 7070.310307] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7070.330231] LustreError: 136028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7070.717916] Lustre: server umount lustre-MDT0001 complete [ 7083.836922] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 7086.827849] Key type lgssc unregistered [ 7087.111865] LNet: 147232:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7087.124118] LNetError: 147232:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7087.138699] LNet: Removed LNI 192.168.204.146@tcp [ 7088.176303] Key type .llcrypt unregistered [ 7088.186394] Key type ._llcrypt unregistered [ 7109.017155] Key type ._llcrypt registered [ 7109.020643] Key type .llcrypt registered [ 7109.162770] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 7128.335187] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 7130.003874] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7130.090874] alg: No test for adler32 (adler32-zlib) [ 7131.234186] Lustre: Lustre: Build Version: 2.17.54_162_gdc75e25 [ 7131.550879] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 7133.346141] Key type lgssc registered [ 7135.061982] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7187.814943] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7187.856044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7189.175822] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7189.264821] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7189.403610] Lustre: lustre-MDT0000: new disk, initializing [ 7189.508940] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7189.546070] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7194.649502] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7207.286082] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7207.453132] Lustre: 150439:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7207.555167] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7207.565871] Lustre: Skipped 1 previous similar message [ 7207.752869] Lustre: lustre-MDT0001: new disk, initializing [ 7207.955838] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7208.095573] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7208.126147] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7213.397610] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7224.411298] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7231.469648] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7239.280330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7239.669696] Lustre: lustre-OST0000: new disk, initializing [ 7239.682239] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7239.688701] Lustre: 152410:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7239.788106] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7245.437168] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7245.449959] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7245.582145] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7246.518524] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7257.262509] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7264.580103] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_module obdecho/obdecho [ 7266.755390] LustreError: 153412:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 7266.760916] LustreError: 153412:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 7269.025677] LustreError: 153509:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 7284.705597] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7284.716643] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7284.738650] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7286.756792] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7286.759443] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7286.784476] Lustre: Skipped 1 previous similar message [ 7286.794400] Lustre: Skipped 2 previous similar messages [ 7288.551428] Lustre: server umount lustre-MDT0000 complete [ 7293.148438] LustreError: 150430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784192144 with bad export cookie 2366041757271433657 [ 7293.162442] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7293.166447] LustreError: 150430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7293.595958] Lustre: server umount lustre-MDT0001 complete [ 7308.869523] Lustre: server umount lustre-OST0000 complete [ 7334.397151] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 7346.965646] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 7398.467414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7398.834301] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7398.888519] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7398.959881] Lustre: lustre-MDT0000: new disk, initializing [ 7399.018182] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7399.045870] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7403.572812] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7414.677908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7414.816596] Lustre: 157232:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7414.870405] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7414.874987] Lustre: Skipped 1 previous similar message [ 7414.975444] Lustre: lustre-MDT0001: new disk, initializing [ 7415.074126] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7415.113251] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7415.128152] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7420.178316] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7430.269433] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7436.845539] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7445.677613] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7446.084046] Lustre: lustre-OST0000: new disk, initializing [ 7446.089081] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7446.112875] Lustre: 159201:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7446.256967] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7453.798424] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7454.358144] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7454.372714] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7454.538547] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7465.460186] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7471.154574] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7471.416862] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7477.533865] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7477.863436] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7480.294295] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7480.300569] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7480.336466] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7485.409374] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7485.420191] Lustre: Skipped 2 previous similar messages [ 7486.331129] Lustre: server umount lustre-OST0000 complete [ 7492.065925] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7492.072580] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7492.080335] Lustre: Skipped 1 previous similar message [ 7492.085233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7497.417840] Lustre: server umount lustre-MDT0000 complete [ 7500.775947] LustreError: 157242:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7500.795675] LustreError: 157242:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 7501.544975] LustreError: 157223:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784192353 with bad export cookie 2366041757271434616 [ 7501.547687] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7501.556827] LustreError: 157223:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7501.943061] Lustre: server umount lustre-MDT0001 complete [ 7505.791557] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 04:59:15 (1784192355) [ 7507.511125] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7509.915611] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 04:59:19 (1784192359) [ 7518.450485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7518.986724] LustreError: 161613:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7519.045205] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7524.328493] LustreError: 161614:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7524.364051] LustreError: 161614:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7525.151656] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7528.419922] LustreError: 161613:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7533.538507] LustreError: 161614:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7534.410390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7539.963299] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7549.446359] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7555.694641] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7562.799872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7563.230821] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7563.240728] Lustre: Skipped 1 previous similar message [ 7569.986547] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7577.733855] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7589.858922] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7589.868282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7589.886993] Lustre: Skipped 2 previous similar messages [ 7589.912981] Lustre: Skipped 4 previous similar messages [ 7595.764276] Lustre: server umount lustre-MDT0000 complete [ 7598.732321] LustreError: 161610:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7600.572345] LustreError: 161596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784192452 with bad export cookie 2366041757271435498 [ 7600.580452] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7600.585995] LustreError: 161596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7605.224425] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7605.245936] Lustre: Skipped 1 previous similar message [ 7607.014278] Lustre: server umount lustre-MDT0001 complete [ 7616.225765] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7631.635303] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7631.863321] LustreError: 165468:0:(llog.c:1655:llog_backup()) MGC192.168.204.146@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7631.880366] Lustre: 165468:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.146@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7634.432956] LustreError: 165473:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7634.457978] LustreError: 165473:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 7666.659408] LustreError: 165474:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7666.698555] LustreError: 165474:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 24 previous similar messages [ 7692.768308] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7692.773484] Lustre: 165468:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.204.146@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7730.817598] LustreError: 165475:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7730.839446] LustreError: 165475:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 39 previous similar messages [ 7733.756800] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7738.863529] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7738.929557] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7749.622713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7762.423706] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 7776.435841] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7777.268840] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7784.198427] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7786.059657] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7792.095739] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7793.930806] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7818.210200] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7818.213663] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7818.231258] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7818.237164] Lustre: Skipped 1 previous similar message [ 7818.246492] Lustre: Skipped 7 previous similar messages [ 7822.550800] Lustre: server umount lustre-MDT0000 complete [ 7827.202728] LustreError: 161596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784192679 with bad export cookie 2366041757271441987 [ 7827.214945] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7827.218489] LustreError: 161596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7828.057520] Lustre: server umount lustre-MDT0001 complete [ 7844.419647] Lustre: server umount lustre-OST0000 complete [ 7869.155847] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 7881.906569] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 7920.023920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7920.321237] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7920.346097] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7920.421711] Lustre: lustre-MDT0000: new disk, initializing [ 7920.607298] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7920.610530] Lustre: Skipped 1 previous similar message [ 7920.662835] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7925.259934] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7937.215437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7937.393514] Lustre: 170578:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7937.678952] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7944.439135] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7955.283910] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7962.723858] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7970.783730] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7971.127652] Lustre: lustre-OST0000: new disk, initializing [ 7971.131267] Lustre: Skipped 1 previous similar message [ 7971.136847] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7971.141465] Lustre: Skipped 2 previous similar messages [ 7971.150132] Lustre: 172547:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7977.053594] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7977.065560] Lustre: Skipped 1 previous similar message [ 7977.071472] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7977.246752] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7979.351962] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7991.331713] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7998.212128] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7998.646684] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8004.629897] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8005.002479] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8008.168149] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8008.170823] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8008.180612] LustreError: Skipped 1 previous similar message [ 8008.186212] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8008.237067] Lustre: Skipped 2 previous similar messages [ 8013.621568] Lustre: server umount lustre-OST0000 complete [ 8032.224201] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8032.456476] Lustre: server umount lustre-MDT0000 complete [ 8033.764880] LustreError: 170586:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8033.795098] LustreError: 170586:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 41 previous similar messages [ 8036.730941] LustreError: 170570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784192888 with bad export cookie 2366041757271458794 [ 8036.738421] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8036.744139] LustreError: 170570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8037.356795] Lustre: server umount lustre-MDT0001 complete [ 8046.777908] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 05:08:16 (1784192896) [ 8053.400862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8058.167514] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8066.912913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8071.974878] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8079.316867] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8085.533645] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8092.882507] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8099.794414] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8107.835493] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8117.948331] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 05:09:27 (1784192967) [ 8124.385107] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8124.387713] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8124.395182] LustreError: Skipped 1 previous similar message [ 8124.395887] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8124.420480] Lustre: Skipped 3 previous similar messages [ 8124.423685] Lustre: Skipped 9 previous similar messages [ 8128.000782] Lustre: server umount lustre-OST0000 complete [ 8138.619426] Lustre: server umount lustre-MDT0000 complete [ 8142.607778] LustreError: 174942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784192994 with bad export cookie 2366041757271459676 [ 8142.617358] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8143.012858] Lustre: server umount lustre-MDT0001 complete [ 8153.453349] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 8155.954349] Key type lgssc unregistered [ 8156.254833] LNet: 178691:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8156.264044] LNetError: 178691:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8156.284298] LNet: Removed LNI 192.168.204.146@tcp [ 8156.978740] Key type .llcrypt unregistered [ 8156.982230] Key type ._llcrypt unregistered [ 8175.417221] Key type ._llcrypt registered [ 8175.419887] Key type .llcrypt registered [ 8175.511209] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 8176.352041] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8176.395677] alg: No test for adler32 (adler32-zlib) [ 8177.452511] Lustre: Lustre: Build Version: 2.17.54_162_gdc75e25 [ 8177.638810] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 8179.305496] Key type lgssc registered [ 8180.718631] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8188.721768] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8188.741397] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8204.321825] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8209.440792] LustreError: 179934:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.146@tcp: failed processing log, type 4: rc = -110 [ 8214.560483] LustreError: 179934:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8214.573510] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 8214.584203] LustreError: MGC192.168.204.146@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 8214.598232] LustreError: 179934:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 8214.610387] LustreError: 179934:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 8214.620610] LustreError: 179934:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 8214.702829] Lustre: server umount lustre-OST0000 complete [ 8214.706299] LustreError: 179934:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8225.654309] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8227.285921] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8231.541195] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8237.539580] LustreError: 180433:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8237.561959] LustreError: 180433:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8241.721261] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8242.048904] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8246.503859] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8254.769186] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8261.272711] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8268.161570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8268.445672] LustreError: 182251:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8268.451836] Lustre: 182251:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.204.146@tcp: invalid local config log lustre-OST0000: rc = -22 [ 8268.560654] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8274.975298] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8282.819472] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8286.251241] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8286.259898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8286.269261] Lustre: Skipped 2 previous similar messages [ 8286.287549] Lustre: Skipped 2 previous similar messages [ 8291.300208] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8291.316805] Lustre: Skipped 2 previous similar messages [ 8291.494225] Lustre: server umount lustre-MDT0000 complete [ 8295.941564] LustreError: 181167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784193147 with bad export cookie 7821458986737483791 [ 8295.945692] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8295.970494] LustreError: 181167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8296.418771] LustreError: 182275:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8296.436509] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8296.492769] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8296.505203] LustreError: 181162:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000036681c96 x1870861989606528/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8296.954275] Lustre: server umount lustre-MDT0001 complete [ 8303.507669] Lustre: server umount lustre-OST0000 complete [ 8319.614565] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 8338.984621] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 8379.848938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8380.064939] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8380.111428] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8380.240903] Lustre: lustre-MDT0000: new disk, initializing [ 8380.356781] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8380.377372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8385.296780] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8396.370865] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8396.488250] Lustre: 185801:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8396.537567] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8396.543832] Lustre: Skipped 1 previous similar message [ 8396.636949] Lustre: lustre-MDT0001: new disk, initializing [ 8396.723871] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8396.754162] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8396.765357] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8401.974300] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8411.572587] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8416.390284] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8421.460595] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8421.654243] Lustre: lustre-OST0000: new disk, initializing [ 8421.658836] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8421.667188] Lustre: 187765:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8421.752900] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8426.170632] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8432.078425] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8432.157592] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8432.165615] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8432.234767] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8434.202063] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8434.331128] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8436.513575] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8436.627263] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8442.339168] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8442.341094] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8442.346409] LustreError: Skipped 1 previous similar message [ 8442.362523] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8442.369720] Lustre: Skipped 1 previous similar message [ 8443.826695] Lustre: server umount lustre-OST0000 complete [ 8452.579935] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8452.582935] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8452.616617] Lustre: Skipped 2 previous similar messages [ 8452.635282] Lustre: Skipped 1 previous similar message [ 8462.826983] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8462.833024] Lustre: Skipped 3 previous similar messages [ 8463.328155] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8463.439748] Lustre: server umount lustre-MDT0000 complete [ 8467.789354] LustreError: 185794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784193319 with bad export cookie 7821458986737484505 [ 8467.809685] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8467.816194] LustreError: 185794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8468.208372] Lustre: server umount lustre-MDT0001 complete [ 8479.732078] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 05:15:29 (1784193329) [ 8499.443394] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 8502.644827] Key type lgssc unregistered [ 8502.975811] LNet: 190709:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8502.993492] LNetError: 190709:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8503.027278] LNet: Removed LNI 192.168.204.146@tcp [ 8503.979705] Key type .llcrypt unregistered [ 8503.985233] Key type ._llcrypt unregistered [ 8524.336452] Key type ._llcrypt registered [ 8524.339989] Key type .llcrypt registered [ 8524.513890] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 8526.111952] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8526.162227] alg: No test for adler32 (adler32-zlib) [ 8527.311870] Lustre: Lustre: Build Version: 2.17.54_162_gdc75e25 [ 8527.708514] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 8529.416230] Key type lgssc registered [ 8530.601761] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8539.461368] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8539.488899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8541.037893] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8545.934363] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8550.375289] LustreError: 191927:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8550.395433] LustreError: 191927:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8555.491949] LustreError: 191928:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8556.379585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8556.842292] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8561.860179] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8569.840403] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8576.061222] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8584.705048] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8584.955458] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8584.959887] LustreError: 193794:0:(llog.c:1696:llog_backup()) MGC192.168.204.146@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8584.973642] Lustre: 193794:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.204.146@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8585.116988] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8592.309280] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8600.875214] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8606.700887] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8606.703063] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8606.718176] Lustre: Skipped 2 previous similar messages [ 8606.724865] Lustre: Skipped 1 previous similar message [ 8609.519117] Lustre: server umount lustre-MDT0000 complete [ 8611.814118] LustreError: 192657:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8611.844177] LustreError: 192657:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8613.458286] LustreError: 192665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784193465 with bad export cookie 4753221494087812626 [ 8613.464483] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8613.475350] LustreError: 192665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8613.905125] Lustre: server umount lustre-MDT0001 complete [ 8628.874819] Lustre: server umount lustre-OST0000 complete [ 8639.922590] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 8651.695630] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 8699.324880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8699.870274] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8699.906787] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8700.003729] Lustre: lustre-MDT0000: new disk, initializing [ 8700.068084] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8700.085969] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8704.963775] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8717.449877] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8717.578385] Lustre: 197324:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8717.661829] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8717.667518] Lustre: Skipped 1 previous similar message [ 8717.671910] LustreError: 198106:0:(llog.c:1696:llog_backup()) MGC192.168.204.146@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8717.679359] LustreError: 198106:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8717.685876] Lustre: 198106:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.146@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8717.735879] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8717.738812] Lustre: Skipped 1 previous similar message [ 8717.825789] Lustre: lustre-MDT0001: new disk, initializing [ 8717.968452] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8718.008359] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8718.026227] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8723.094824] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8732.675323] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8739.524789] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8746.371350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8746.646663] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8746.649487] Lustre: Skipped 1 previous similar message [ 8746.652334] LustreError: 199268:0:(llog.c:1696:llog_backup()) MGC192.168.204.146@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8746.661778] LustreError: 199268:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8746.666938] Lustre: 199268:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.146@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8746.676306] Lustre: 199268:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8746.705917] Lustre: lustre-OST0000: new disk, initializing [ 8746.710418] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8746.718412] Lustre: 199291:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8746.804688] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8753.399683] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8754.268425] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8754.288409] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8754.458407] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8763.986689] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8769.447497] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8769.736562] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8775.261754] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8775.572413] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8780.258753] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8780.264586] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8780.265873] LustreError: Skipped 1 previous similar message [ 8780.270125] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8780.270134] Lustre: Skipped 1 previous similar message [ 8780.315099] Lustre: Skipped 1 previous similar message [ 8784.414352] Lustre: server umount lustre-OST0000 complete [ 8789.988994] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8789.998248] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8790.022575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8790.030872] Lustre: Skipped 1 previous similar message [ 8795.622193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8795.628779] Lustre: Skipped 3 previous similar messages [ 8795.891873] Lustre: server umount lustre-MDT0000 complete [ 8800.104755] LustreError: 197315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784193651 with bad export cookie 4753221494087813347 [ 8800.106134] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8800.130235] LustreError: 197315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8800.480828] Lustre: server umount lustre-MDT0001 complete [ 8809.901496] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 05:20:59 (1784193659) [ 8829.844732] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 8840.651413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8841.192527] LustreError: 202419:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8841.245320] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8846.234766] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8846.308641] LustreError: 202420:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8855.931722] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8861.752616] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8865.234517] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8873.878637] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8874.124839] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8874.133325] Lustre: Skipped 1 previous similar message [ 8881.598792] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8891.338103] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8891.485196] Lustre: lustre-OST0001: new disk, initializing [ 8891.491773] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8891.510707] Lustre: 204727:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8897.673979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8897.703032] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8897.866820] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8898.320778] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8911.880748] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8918.940685] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8921.915359] Lustre: DEBUG MARKER: ADD OST3 [ 8929.939048] loop: module loaded [ 8930.053396] loop0: detected capacity change from 0 to 204800000 [ 8930.309396] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8930.334149] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8930.350103] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8937.738795] loop0: detected capacity change from 0 to 204800000 [ 8938.309347] Lustre: DEBUG MARKER: STOP OST3 [ 8944.566607] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8944.822677] Lustre: lustre-OST0003: new disk, initializing [ 8944.827270] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8944.838456] Lustre: 206766:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8944.970832] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8944.977109] Lustre: Skipped 1 previous similar message [ 8946.033992] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8946.222436] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8946.230394] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8946.278510] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8948.065868] LustreError: 206867:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8948.688101] Lustre: Failing over lustre-OST0003 [ 8948.881587] Lustre: server umount lustre-OST0003 complete [ 8951.277118] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8951.288813] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8951.305166] Lustre: Skipped 2 previous similar messages [ 8951.306217] LustreError: 204735:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8951.333363] LustreError: 204735:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8953.173169] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8962.500687] Lustre: DEBUG MARKER: START OST3 again [ 8970.050636] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8970.531780] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8971.895308] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8972.142202] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8972.148298] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8978.857191] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9002.761101] Lustre: Failing over lustre-OST0003 [ 9002.934846] Lustre: server umount lustre-OST0003 complete [ 9002.981741] LustreError: lustre-OST0003-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9002.983789] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9002.992833] LustreError: Skipped 1 previous similar message [ 9002.994965] LustreError: 203904:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9002.994980] LustreError: 203904:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 9003.069976] Lustre: Skipped 1 previous similar message [ 9025.506599] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9025.517586] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9025.537693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9028.677373] Lustre: server umount lustre-MDT0000 complete [ 9033.438094] LustreError: 203154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784193885 with bad export cookie 4753221494087814229 [ 9033.450213] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9033.452882] LustreError: 203154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9037.286961] LustreError: 208163:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9037.290278] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9037.335957] LustreError: 208163:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 16 previous similar messages [ 9037.366252] Lustre: Skipped 5 previous similar messages [ 9040.006344] Lustre: server umount lustre-MDT0001 complete [ 9044.173493] Lustre: server umount lustre-OST0000 complete [ 9049.217377] Lustre: server umount lustre-OST0001 complete [ 9057.520707] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 9068.169911] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 9111.384153] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9111.618911] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9111.642370] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9111.699593] Lustre: lustre-MDT0000: new disk, initializing [ 9111.759995] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9111.770930] Lustre: Skipped 1 previous similar message [ 9111.785271] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9116.787229] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9127.975350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9128.130309] Lustre: 212194:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9128.185296] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9128.188827] Lustre: Skipped 1 previous similar message [ 9128.256516] Lustre: lustre-MDT0001: new disk, initializing [ 9128.387590] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9128.397799] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9132.917427] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9142.555550] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9148.351206] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9156.192225] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9156.435421] Lustre: 214163:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9158.089263] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9158.223927] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9163.073711] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9174.161439] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9180.748849] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9181.068616] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9185.836975] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9186.082588] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9188.842278] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9188.848893] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9188.857748] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9188.857759] Lustre: Skipped 2 previous similar messages [ 9188.858202] Lustre: Skipped 6 previous similar messages [ 9188.884867] LustreError: Skipped 1 previous similar message [ 9194.427669] Lustre: server umount lustre-OST0000 complete [ 9204.833337] Lustre: server umount lustre-MDT0000 complete [ 9208.634204] LustreError: 212186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784194060 with bad export cookie 4753221494087816525 [ 9208.640062] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9208.645882] LustreError: 212186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9208.956559] Lustre: server umount lustre-MDT0001 complete [ 9212.955186] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 05:27:42 (1784194062) [ 9232.555650] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 9244.821718] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 9283.138054] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9283.439933] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9283.462452] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9283.467537] Lustre: Skipped 1 previous similar message [ 9283.547527] Lustre: lustre-MDT0000: new disk, initializing [ 9283.550717] Lustre: Skipped 1 previous similar message [ 9283.630358] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9283.636546] Lustre: Skipped 1 previous similar message [ 9288.227283] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9299.043693] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9299.165911] Lustre: 218775:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9299.604570] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9306.187467] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9318.241640] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9324.658856] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9333.178352] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9333.690656] Lustre: 220745:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9335.172098] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9335.355226] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9341.421260] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9351.428307] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9356.501584] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9356.680531] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9361.565898] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9361.746138] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9365.992132] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9365.994645] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9365.995283] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9366.009472] Lustre: Skipped 3 previous similar messages [ 9366.031405] Lustre: Skipped 8 previous similar messages [ 9378.272311] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9378.466870] Lustre: server umount lustre-OST0000 complete [ 9389.106952] Lustre: server umount lustre-MDT0000 complete [ 9391.585387] LustreError: 218785:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9391.628504] LustreError: 218785:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9392.932338] LustreError: 218765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784194244 with bad export cookie 4753221494087817407 [ 9392.934037] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9392.946944] LustreError: 218765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9393.432971] Lustre: server umount lustre-MDT0001 complete [ 9402.404896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9403.015407] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9403.018675] Lustre: Skipped 5 previous similar messages [ 9408.129501] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9417.268633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9421.977242] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9429.911764] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9436.620234] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9443.481078] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9449.999447] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9457.729844] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9470.949495] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 9470.957163] LustreError: Skipped 2 previous similar messages [ 9470.963084] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9470.989873] Lustre: Skipped 2 previous similar messages [ 9471.000086] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9471.008171] Lustre: Skipped 6 previous similar messages [ 9474.905222] Lustre: server umount lustre-OST0000 complete [ 9486.199685] Lustre: server umount lustre-MDT0000 complete [ 9490.177170] LustreError: 225842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784194342 with bad export cookie 4753221494087818289 [ 9490.183781] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9490.194648] LustreError: 225842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9490.701732] Lustre: server umount lustre-MDT0001 complete [ 9595.639034] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 9598.179942] Key type lgssc unregistered [ 9598.503399] LNet: 226633:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9598.514778] LNetError: 226633:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9598.533172] LNet: Removed LNI 192.168.204.146@tcp [ 9599.246611] Key type .llcrypt unregistered [ 9599.251789] Key type ._llcrypt unregistered [ 9629.976699] Key type ._llcrypt registered [ 9629.987399] Key type .llcrypt registered [ 9630.180661] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 9645.810123] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 9647.053622] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9647.099335] alg: No test for adler32 (adler32-zlib) [ 9648.182496] Lustre: Lustre: Build Version: 2.17.54_162_gdc75e25 [ 9648.464732] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 9650.161338] Key type lgssc registered [ 9651.420793] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9688.896871] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9688.927432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9690.283898] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9690.324510] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9690.433701] Lustre: lustre-MDT0000: new disk, initializing [ 9690.519880] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9690.552553] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9695.467174] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9707.102709] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9707.288383] Lustre: 229988:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9707.375047] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9707.381220] Lustre: Skipped 1 previous similar message [ 9707.485939] Lustre: lustre-MDT0001: new disk, initializing [ 9707.637566] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9707.683747] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9707.705345] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9713.331381] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9724.400522] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9731.722898] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9739.054377] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9739.312920] Lustre: lustre-OST0000: new disk, initializing [ 9739.317339] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9739.324499] Lustre: 231956:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9739.433592] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9745.696712] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9746.012031] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9746.023121] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9746.122841] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9755.368681] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9760.750707] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9760.960318] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9765.661810] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9765.892552] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9772.008054] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9772.014406] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9772.039421] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9772.048472] Lustre: Skipped 1 previous similar message [ 9775.011079] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9775.033661] Lustre: Skipped 1 previous similar message [ 9780.203332] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9780.207179] Lustre: Skipped 1 previous similar message [ 9782.240151] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9782.546230] Lustre: server umount lustre-OST0000 complete [ 9789.413637] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9789.427560] LustreError: Skipped 1 previous similar message [ 9789.431030] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9789.444522] Lustre: Skipped 1 previous similar message [ 9789.452955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9790.437437] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9793.817261] Lustre: server umount lustre-MDT0000 complete [ 9797.780900] LustreError: 229980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784194649 with bad export cookie 14448440603664111339 [ 9797.790224] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9797.791444] LustreError: 229980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9798.294620] Lustre: server umount lustre-MDT0001 complete [ 9820.187990] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 9831.744815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9832.233932] LustreError: 234893:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9832.263017] LustreError: 234893:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9832.290844] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9837.296528] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9837.547384] LustreError: 234894:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9842.682762] LustreError: 234893:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9846.360273] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9846.827425] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9851.633870] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9854.543286] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9860.401426] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9860.696566] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9866.552234] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9876.909467] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9877.084980] Lustre: lustre-OST0001: new disk, initializing [ 9877.092151] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9877.100187] Lustre: 237201:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9877.238438] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9882.881581] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9882.909083] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9883.075152] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9884.917586] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9897.189726] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9906.820254] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9922.181566] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 05:39:31 (1784194771) [ 9928.677762] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9928.686238] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9928.697054] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9928.699833] Lustre: Skipped 2 previous similar messages [ 9932.597407] Lustre: server umount lustre-MDT0000 complete [ 9934.322290] LustreError: 234893:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9934.340484] LustreError: 234893:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9937.328815] LustreError: 235630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784194789 with bad export cookie 14448440603664112214 [ 9937.336313] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9937.343365] LustreError: 235630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9937.863972] Lustre: server umount lustre-MDT0001 complete [ 9952.901672] Lustre: server umount lustre-OST0000 complete [ 9955.874693] Lustre: 228209:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784194791/real 1784194791] req@ffff93ce82389180 x1870863532150016/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784194807 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9955.915824] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9955.932506] Lustre: Skipped 3 previous similar messages [ 9957.494615] Lustre: server umount lustre-OST0001 complete [ 9965.394849] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 9976.118576] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [10017.915837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10018.136481] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10018.183768] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10018.257369] Lustre: lustre-MDT0000: new disk, initializing [10018.323343] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10018.347555] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10023.678975] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10036.060813] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10036.204725] Lustre: 241152:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10036.247082] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10036.250767] Lustre: Skipped 1 previous similar message [10036.315355] Lustre: lustre-MDT0001: new disk, initializing [10036.495804] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10036.506447] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10041.234922] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10051.504634] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10057.409741] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10064.558029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10064.809224] Lustre: lustre-OST0000: new disk, initializing [10064.823552] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10064.837452] Lustre: 243123:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10066.625189] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10066.640928] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10066.769380] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10071.932469] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10082.470855] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10088.500307] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10088.781497] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10093.868723] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10094.100380] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10097.639994] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10097.647602] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10097.660684] Lustre: Skipped 1 previous similar message [10097.673855] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10097.685026] Lustre: Skipped 5 previous similar messages [10102.523438] Lustre: server umount lustre-OST0000 complete [10112.371473] Lustre: server umount lustre-MDT0000 complete [10112.993838] LustreError: 241159:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10113.016873] LustreError: 241159:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10115.991402] LustreError: 241146:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784194967 with bad export cookie 14448440603664114356 [10116.014562] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10116.015223] LustreError: 241146:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10116.471357] Lustre: server umount lustre-MDT0001 complete [10124.812440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10125.254647] LustreError: 245338:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10125.321119] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10125.326289] Lustre: Skipped 2 previous similar messages [10130.306465] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10139.111957] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10144.133201] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10151.282159] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10156.840632] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10163.619478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10169.627816] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10177.072602] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10323.944653] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10323.955728] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10323.959786] Lustre: Skipped 3 previous similar messages [10323.961085] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10323.961544] LustreError: Skipped 1 previous similar message [10323.988939] Lustre: Skipped 3 previous similar messages [10337.250451] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10337.466847] Lustre: server umount lustre-OST0000 complete [10344.433599] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10348.069408] Lustre: server umount lustre-MDT0000 complete [10349.543365] LustreError: 247243:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10349.573187] LustreError: 247243:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10352.249548] LustreError: 245319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784195204 with bad export cookie 14448440603664115238 [10352.256774] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10352.270717] LustreError: 245319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10352.567515] Lustre: server umount lustre-MDT0001 complete [10448.614898] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [10451.633850] Key type lgssc unregistered [10452.049472] LNet: 248950:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10452.060935] LNetError: 248950:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10452.090252] LNet: Removed LNI 192.168.204.146@tcp [10453.069595] Key type .llcrypt unregistered [10453.074085] Key type ._llcrypt unregistered [10480.739616] Key type ._llcrypt registered [10480.742737] Key type .llcrypt registered [10480.907623] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [10498.411825] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [10499.646342] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10499.674939] alg: No test for adler32 (adler32-zlib) [10500.746338] Lustre: Lustre: Build Version: 2.17.54_162_gdc75e25 [10501.053084] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [10502.768191] Key type lgssc registered [10504.133819] Lustre: Echo OBD driver; http://www.lustre.org/ [10552.605707] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [10564.016505] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10564.052591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10565.363725] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10565.393286] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10565.468366] Lustre: lustre-MDT0000: new disk, initializing [10565.585818] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10565.635226] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10570.466380] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10582.819215] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10582.981711] Lustre: 253023:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10583.077412] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10583.084972] Lustre: Skipped 1 previous similar message [10583.157665] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [10583.192085] Lustre: lustre-MDT0001: new disk, initializing [10583.287931] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10583.334932] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10583.352565] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10588.720848] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10593.982835] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10602.033628] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10602.466322] Lustre: lustre-OST0000: new disk, initializing [10602.474653] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10602.487114] Lustre: 254653:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10602.624358] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10607.176094] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10607.188366] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10607.364381] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10610.336408] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10623.364388] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10623.575417] Lustre: lustre-OST0001: new disk, initializing [10623.579757] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10623.589498] Lustre: 255525:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10623.691078] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10630.499762] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10632.818713] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10632.848152] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10632.952951] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10641.753291] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10648.972949] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10663.911835] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10663.914125] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10663.932621] Lustre: Skipped 3 previous similar messages [10663.940587] Lustre: Skipped 2 previous similar messages [10668.327917] Lustre: server umount lustre-MDT0000 complete [10672.543198] LustreError: 253013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784195524 with bad export cookie 13457316110161630548 [10672.557381] LustreError: 253013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10672.561200] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10672.862150] Lustre: server umount lustre-MDT0001 complete [10686.303887] Lustre: server umount lustre-OST0000 complete [10689.315245] Lustre: 250528:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784195525/real 1784195525] req@ffff93ce81aa1180 x1870864425497600/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784195541 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10689.372716] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10690.393727] Lustre: server umount lustre-OST0001 complete [10705.081247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10715.282684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10725.770259] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10737.823529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10763.920458] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [10775.934050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10775.961526] Lustre: lustre-MDT0000: reset Object Index mappings [10776.508755] LustreError: 260540:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10776.543858] LustreError: 260540:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [10776.690826] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10781.677659] LustreError: 260541:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10781.931478] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10786.789327] LustreError: 260540:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10791.665938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10791.695895] Lustre: lustre-MDT0001: reset Object Index mappings [10791.910707] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [10791.917688] Lustre: Skipped 1 previous similar message [10792.127551] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10797.542283] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10801.118979] Lustre: 261682:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10809.252898] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10809.739991] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10816.887527] LustreError: 262037:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10817.164267] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10822.002424] LustreError: 262037:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10822.016501] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10822.028258] LustreError: 262037:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10826.432139] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10831.894867] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10834.731907] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10843.313177] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10847.383531] Lustre: 263555:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10853.347257] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10853.357840] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10853.375850] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10854.869071] Lustre: server umount lustre-MDT0000 complete [10857.450410] LustreError: 260540:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10857.456974] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10857.482767] Lustre: Skipped 1 previous similar message [10858.725605] LustreError: 260520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784195710 with bad export cookie 13457316110161637338 [10858.738222] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10858.743061] LustreError: 260520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10859.086470] Lustre: server umount lustre-MDT0001 complete [10873.855973] Lustre: server umount lustre-OST0000 complete [10888.956843] Lustre: server umount lustre-OST0001 complete [10913.317180] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 05:56:03 (1784195763) [10933.332240] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [10945.849317] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [10982.262398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10982.483760] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10982.516374] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10982.585883] Lustre: lustre-MDT0000: new disk, initializing [10982.642806] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10982.646578] Lustre: Skipped 1 previous similar message [10982.656510] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10986.274489] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10995.675565] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10995.789828] Lustre: 267812:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10995.843377] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10995.846410] Lustre: Skipped 1 previous similar message [10995.913570] Lustre: lustre-MDT0001: new disk, initializing [10996.055599] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10996.067334] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11000.628365] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11010.030260] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11016.383444] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11023.822977] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11024.065579] Lustre: lustre-OST0000: new disk, initializing [11024.068844] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11024.074548] Lustre: 269781:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11025.253139] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11025.266854] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11025.446421] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11030.313827] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11039.442543] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11043.919154] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11044.085000] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11048.631069] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11048.959641] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11050.981815] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11050.984489] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11050.986448] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11050.992733] LustreError: Skipped 1 previous similar message [11051.018259] Lustre: Skipped 2 previous similar messages [11056.826515] Lustre: server umount lustre-OST0000 complete [11061.222765] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11061.224932] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11061.231633] Lustre: Skipped 1 previous similar message [11061.235980] Lustre: Skipped 3 previous similar messages [11066.762558] Lustre: server umount lustre-MDT0000 complete [11070.461801] LustreError: 267803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784195922 with bad export cookie 13457316110161639431 [11070.466503] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11070.472047] LustreError: 267803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11070.969810] Lustre: server umount lustre-MDT0001 complete [11094.072606] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [11105.724382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11106.220940] LustreError: 272727:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11106.242153] LustreError: 272727:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11106.277231] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11106.291513] Lustre: Skipped 2 previous similar messages [11110.606389] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11118.946342] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11123.214852] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11126.176837] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11132.563788] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11139.525418] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11147.882704] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11148.091067] Lustre: lustre-OST0001: new disk, initializing [11148.096437] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11148.104730] Lustre: 275034:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11149.511641] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11149.519716] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11149.658152] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11155.084926] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11164.539069] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11168.159625] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11172.863272] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [11173.079489] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [11182.579992] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11182.589962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11182.602257] Lustre: Skipped 3 previous similar messages [11182.616419] Lustre: Skipped 5 previous similar messages [11187.502241] Lustre: server umount lustre-MDT0000 complete [11187.686342] LustreError: 272728:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11187.711846] LustreError: 272728:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [11190.879973] LustreError: 273462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784196042 with bad export cookie 13457316110161640313 [11190.886849] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11190.895403] LustreError: 273462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11191.301725] Lustre: server umount lustre-MDT0001 complete [11205.863929] Lustre: server umount lustre-OST0000 complete [11208.929099] Lustre: 250529:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784196044/real 1784196044] req@ffff93cf83802680 x1870864425657728/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784196060 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11210.217268] Lustre: server umount lustre-OST0001 complete [11228.164909] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [11237.946355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11238.350280] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11238.360178] Lustre: Skipped 3 previous similar messages [11242.131283] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11249.659154] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11253.811235] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11256.488354] Lustre: 279241:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11263.069043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11264.369834] LustreError: 279596:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11264.393427] LustreError: 279596:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11264.423089] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [11269.361816] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11278.719599] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11284.490579] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [11287.054851] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11297.533679] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11301.066720] Lustre: 281116:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11306.492866] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [11306.515713] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [11316.296077] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 06:02:46 (1784196166) [11332.065149] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11332.071689] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11332.089819] Lustre: Skipped 1 previous similar message [11332.097082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11337.818699] Lustre: server umount lustre-MDT0000 complete [11341.940228] LustreError: 278082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784196193 with bad export cookie 13457316110161646921 [11341.956186] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11341.963992] LustreError: 278082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11348.440221] Lustre: server umount lustre-MDT0001 complete [11358.780890] Lustre: server umount lustre-OST0000 complete [11363.684348] Lustre: server umount lustre-OST0001 complete [11371.399782] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [11382.497307] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [11437.758824] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [11449.140578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11449.375588] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11449.413472] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11449.502979] Lustre: lustre-MDT0000: new disk, initializing [11449.599749] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11454.961216] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11467.595386] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11467.699819] Lustre: 285360:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11467.952624] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11474.861456] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11489.712311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11490.228626] Lustre: 286993:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11491.920974] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11497.513053] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11508.957486] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11509.186692] Lustre: 287864:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11509.368353] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11509.379656] Lustre: Skipped 6 previous similar messages [11510.993516] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11511.006891] Lustre: Skipped 1 previous similar message [11511.097877] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11516.578907] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11527.201604] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11531.316687] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11531.320515] Lustre: Skipped 1 previous similar message [11548.289958] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 06:06:37 (1784196397) [11552.236993] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11552.249282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11552.261469] Lustre: Skipped 8 previous similar messages [11552.269282] Lustre: Skipped 10 previous similar messages [11552.749986] Lustre: server umount lustre-MDT0000 complete [11556.033693] LustreError: 286155:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784196407 with bad export cookie 13457316110161654558 [11556.040536] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11556.057202] LustreError: 286155:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11556.342974] Lustre: server umount lustre-MDT0001 complete [11571.016878] Lustre: server umount lustre-OST0000 complete [11573.201210] Lustre: 250529:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784196409/real 1784196409] req@ffff93cfad97a680 x1870864425812736/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784196425 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11575.290446] Lustre: server umount lustre-OST0001 complete [11584.902416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11585.624030] LustreError: 290113:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11585.641335] LustreError: 290113:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [11592.776302] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11604.483876] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11610.018841] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11618.481066] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11624.045259] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11630.679065] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11637.201634] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11644.727810] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11673.333634] loop1: detected capacity change from 0 to 204800000 [11673.753857] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11673.772745] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11673.808257] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11679.506169] loop1: detected capacity change from 0 to 204800000 [11686.497291] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11686.610902] LustreError: 290102:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs3104: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11686.636342] LustreError: 290102:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs3104 [11686.651994] LustreError: 290102:0:(mgs_handler.c:517:mgs_target_reg()) fs3104-MDT0001: Error checking target index: rc = -13 [11686.671906] LustreError: fs3104-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11686.696799] LustreError: 293855:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11686.707773] LustreError: 293855:0:(tgt_mount.c:2129:server_put_super()) no obd fs3104-MDT0001 [11686.714930] LustreError: 293855:0:(tgt_mount.c:116:server_deregister_mount()) fs3104-MDT0001 not registered [11686.772811] Lustre: server umount fs3104-MDT0001 complete [11686.777794] LustreError: 293855:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [11697.129929] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11697.353931] Lustre: Setting parameter fs3104-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs3104-MDT0001 [11697.444406] Lustre: srv-fs3104-MDT0001: No data found on store. Initialize space. [11697.453487] Lustre: Skipped 4 previous similar messages [11697.621588] Lustre: fs3104-MDT0001: new disk, initializing [11697.626073] Lustre: Skipped 3 previous similar messages [11697.787823] Lustre: srv-fs3104-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11697.805117] Lustre: 294322:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs3104-MDT0001-mdtlov: can't create new llog: rc = -115 [11697.820566] LustreError: 294322:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs3104-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11702.843319] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11710.787372] Lustre: server umount fs3104-MDT0001 complete [11726.795444] loop1: detected capacity change from 0 to 204800000 [11727.342112] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11732.454265] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11733.187502] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11733.560923] Lustre: 295890:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs3104-MDT0001-mdtlov: can't create new llog: rc = -115 [11733.580247] LustreError: 295890:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs3104-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11738.720676] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11743.541395] Lustre: server umount fs3104-MDT0001 complete [11758.055670] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11758.059278] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11758.101679] Lustre: Skipped 2 previous similar messages [11761.471577] Lustre: server umount lustre-MDT0000 complete [11771.032349] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11771.205669] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11776.505434] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11777.642294] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11788.290546] loop1: detected capacity change from 0 to 204800000 [11794.713245] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11795.266181] Lustre: 298336:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs3104-MDT0001-mdtlov: can't create new llog: rc = -115 [11795.274733] LustreError: 298336:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs3104-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11801.141386] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11806.459549] Lustre: server umount fs3104-MDT0001 complete [11832.802633] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11837.599588] Lustre: server umount lustre-OST0000 complete [11848.655735] Lustre: server umount lustre-MDT0000 complete [11852.773377] LustreError: 290113:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11852.795697] LustreError: 290113:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 117 previous similar messages [11853.051151] LustreError: 290095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784196704 with bad export cookie 13457316110161657239 [11853.056173] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11853.061670] LustreError: 290095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11853.629988] Lustre: server umount lustre-MDT0001 complete [11866.890814] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [11870.058892] Key type lgssc unregistered [11870.469750] LNet: 300682:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11870.479957] LNetError: 300682:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11870.499426] LNet: Removed LNI 192.168.204.146@tcp [11871.338216] Key type .llcrypt unregistered [11871.340133] Key type ._llcrypt unregistered [11894.551823] Key type ._llcrypt registered [11894.553882] Key type .llcrypt registered [11894.670841] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [11896.100818] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11896.138575] alg: No test for adler32 (adler32-zlib) [11897.316057] Lustre: Lustre: Build Version: 2.17.54_162_gdc75e25 [11897.531298] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [11899.216186] Key type lgssc registered [11900.360747] Lustre: Echo OBD driver; http://www.lustre.org/ [11908.327051] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11908.374621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11909.901916] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11914.778451] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11919.340338] LustreError: 301953:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11919.382156] LustreError: 301953:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11923.779496] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11924.331096] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11929.127383] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11937.314679] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11943.501444] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11949.982582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11950.065218] LustreError: 303784:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11950.407294] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11954.561445] LustreError: 303785:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11954.586465] LustreError: 303785:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11956.082291] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11959.786430] LustreError: 304042:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11959.801143] LustreError: 304042:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11963.426605] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11964.903454] LustreError: 303786:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11964.926726] LustreError: 303786:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11975.150127] LustreError: 304042:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11975.175746] LustreError: 304042:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [11981.221251] loop1: detected capacity change from 0 to 204800000 [11981.413222] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11981.431224] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11981.467611] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11986.726794] loop1: detected capacity change from 0 to 204800000 [11991.701753] LustreError: 303785:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11991.722657] LustreError: 303785:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [11992.956445] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11993.148679] Lustre: fs8512-OST0000: new disk, initializing [11993.158525] Lustre: srv-fs8512-OST0000: No data found on store. Initialize space. [11993.165521] Lustre: 305692:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11993.261986] Lustre: fs8512-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12000.064708] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12008.428790] Lustre: server umount fs8512-OST0000 complete [12021.619307] loop1: detected capacity change from 0 to 204800000 [12026.352530] LustreError: 304041:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12026.372438] LustreError: 304041:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 19 previous similar messages [12027.749363] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12027.991685] Lustre: fs8512-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12034.879571] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12036.587895] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [12044.259471] loop2: detected capacity change from 0 to 204800000 [12044.617150] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12044.629714] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12044.640747] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12049.995925] loop2: detected capacity change from 0 to 204800000 [12055.394057] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12055.486454] LustreError: fs8512-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [12055.500541] LustreError: fs8512-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [12055.508168] LustreError: 308732:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [12055.513048] LustreError: 308732:0:(tgt_mount.c:2129:server_put_super()) no obd fs8512-OST0001 [12055.519687] LustreError: 308732:0:(tgt_mount.c:116:server_deregister_mount()) fs8512-OST0001 not registered [12055.571598] Lustre: server umount fs8512-OST0001 complete [12055.577132] LustreError: 308732:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [12065.483883] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12065.629798] Lustre: fs8512-OST0001: new disk, initializing [12065.632687] Lustre: srv-fs8512-OST0001: No data found on store. Initialize space. [12065.638639] Lustre: 309283:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12065.753865] Lustre: fs8512-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12071.562395] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12080.221240] Lustre: server umount fs8512-OST0000 complete [12089.602121] Lustre: server umount fs8512-OST0001 complete [12092.911443] LustreError: 303786:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12092.919372] LustreError: 303786:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 38 previous similar messages [12102.524389] loop1: detected capacity change from 0 to 204800000 [12108.894451] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12109.109651] Lustre: fs8512-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12116.419661] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12123.311923] loop2: detected capacity change from 0 to 204800000 [12128.621527] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12135.200530] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12143.032206] Lustre: server umount fs8512-OST0000 complete [12151.566523] Lustre: server umount fs8512-OST0001 complete [12164.588102] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12164.589497] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12164.596367] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12164.602586] Lustre: Skipped 1 previous similar message [12164.626052] LustreError: Skipped 2 previous similar messages [12169.701506] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12169.705021] Lustre: Skipped 1 previous similar message [12170.096592] Lustre: server umount lustre-OST0000 complete [12174.837052] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12174.853358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12174.860709] Lustre: Skipped 1 previous similar message [12179.420253] Lustre: server umount lustre-MDT0000 complete [12182.525648] LustreError: 301934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784197034 with bad export cookie 910116822570777823 [12182.526878] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12182.533918] LustreError: 301934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12182.776749] Lustre: server umount lustre-MDT0001 complete [12192.288968] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [12194.799267] Key type lgssc unregistered [12195.063175] LNet: 315204:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12195.073899] LNetError: 315204:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12196.134606] LNet: Removed LNI 192.168.204.146@tcp [12196.984143] Key type .llcrypt unregistered [12196.986529] Key type ._llcrypt unregistered [12224.551726] Key type ._llcrypt registered [12224.556220] Key type .llcrypt registered [12224.671390] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [12238.306437] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [12239.166037] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12239.198590] alg: No test for adler32 (adler32-zlib) [12240.256932] Lustre: Lustre: Build Version: 2.17.54_162_gdc75e25 [12240.466545] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [12242.096207] Key type lgssc registered [12243.164875] Lustre: Echo OBD driver; http://www.lustre.org/ [12278.323931] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12278.364340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12279.622524] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12279.652085] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12279.726001] Lustre: lustre-MDT0000: new disk, initializing [12279.835319] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12279.852052] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12283.872483] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12294.209231] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12294.296624] Lustre: 319078:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12294.338333] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12294.342377] Lustre: Skipped 1 previous similar message [12294.404527] Lustre: lustre-MDT0001: new disk, initializing [12294.491585] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12294.535997] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12294.547696] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12298.754934] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12307.993482] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12312.727986] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12318.221768] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12318.463577] Lustre: lustre-OST0000: new disk, initializing [12318.469477] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12318.474951] Lustre: 321047:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12318.575260] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12323.462466] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12324.457087] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12324.471817] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12324.578347] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12330.775438] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12332.433163] LustreError: 317252:0:(lmv_obd.c:208:lmv_notify()) activation of lustre-MDT0000_UUID failed: -22 [12332.464466] Lustre: Mounted lustre-client [12333.222642] Lustre: Unmounted lustre-client [12342.755579] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12342.763247] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12342.768296] Lustre: Skipped 1 previous similar message [12346.505630] Lustre: server umount lustre-OST0000 complete [12350.945436] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12350.957286] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12350.977323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12350.981657] Lustre: Skipped 1 previous similar message [12352.993457] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12352.996442] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12356.479790] Lustre: server umount lustre-MDT0000 complete [12359.136872] LustreError: 319071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784197210 with bad export cookie 12604451106723121693 [12359.139919] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12359.148732] LustreError: 319071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12359.359748] Lustre: server umount lustre-MDT0001 complete [12367.446139] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [12369.410437] Key type lgssc unregistered [12383.968802] LNet: 323208:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12383.986565] LNetError: 323208:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12461.863612] LNet: Removed LNI 192.168.204.146@tcp [12462.579776] Key type .llcrypt unregistered [12462.581937] Key type ._llcrypt unregistered [12476.980539] Key type ._llcrypt registered [12476.982591] Key type .llcrypt registered [12477.056265] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [12490.654115] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [12491.436514] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12491.528280] alg: No test for adler32 (adler32-zlib) [12492.579228] Lustre: Lustre: Build Version: 2.17.54_162_gdc75e25 [12492.776569] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [12494.464185] Key type lgssc registered [12495.496338] Lustre: Echo OBD driver; http://www.lustre.org/ [12522.016417] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 06:22:52 (1784197372) [12527.200659] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12527.218360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12528.441316] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12528.460240] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12528.523057] Lustre: lustre-MDT0000: new disk, initializing [12528.633650] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12528.656881] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12532.607484] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12541.459740] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12541.595114] Lustre: 326430:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12541.697939] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12541.711752] Lustre: Skipped 1 previous similar message [12541.803427] Lustre: lustre-MDT0001: new disk, initializing [12541.934764] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12541.981682] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12542.000977] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12545.584766] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12554.199528] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12558.931188] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12565.459874] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12565.747118] Lustre: lustre-OST0000: new disk, initializing [12565.752635] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12565.760283] Lustre: 328398:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12565.853619] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12571.275497] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12574.267348] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12574.288284] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12574.378919] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12579.827794] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12595.170159] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12595.174608] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12595.177405] LustreError: Skipped 1 previous similar message [12595.178663] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12595.200155] Lustre: Skipped 1 previous similar message [12595.224182] Lustre: Skipped 1 previous similar message [12598.150537] Lustre: server umount lustre-OST0000 complete [12603.362727] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12603.375905] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12603.395122] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12605.411421] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12605.419493] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12607.581559] Lustre: server umount lustre-MDT0000 complete [12610.895158] LustreError: 327225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784197462 with bad export cookie 5089444072616797429 [12610.899942] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12610.908818] LustreError: 327225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12611.165481] Lustre: server umount lustre-MDT0001 complete [12621.231486] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [12623.242462] Key type lgssc unregistered [12623.481392] LNet: 330305:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12623.493726] LNetError: 330305:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12623.514642] LNet: Removed LNI 192.168.204.146@tcp [12624.165146] Key type .llcrypt unregistered [12624.170981] Key type ._llcrypt unregistered [12643.061123] Key type ._llcrypt registered [12643.065439] Key type .llcrypt registered [12643.157827] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [12644.221711] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12644.323346] alg: No test for adler32 (adler32-zlib) [12645.385560] Lustre: Lustre: Build Version: 2.17.54_162_gdc75e25 [12645.653347] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [12647.368173] Key type lgssc registered [12648.496942] Lustre: Echo OBD driver; http://www.lustre.org/ [12654.562672] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12654.581453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12655.921937] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12659.697736] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12665.325936] LustreError: 331579:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12665.341309] LustreError: 331579:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12668.287154] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12668.759778] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12672.615021] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12680.090753] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12685.779098] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12692.295536] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12692.634398] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12698.208667] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12705.254942] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12714.272574] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12715.737906] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12720.100659] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12720.109127] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12720.146725] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12722.335144] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.46@tcp (stopping) [12722.667686] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12722.687645] Lustre: Skipped 1 previous similar message [12725.482154] Lustre: server umount lustre-MDT0000 complete [12727.427285] LustreError: 333475:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12728.991333] LustreError: 331560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784197580 with bad export cookie 17904980797215330468 [12728.995225] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12729.009167] LustreError: 331560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12732.563274] LustreError: 331574:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12732.584178] LustreError: 331574:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12732.607306] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.46@tcp (stopping) [12732.611045] Lustre: Skipped 3 previous similar messages [12732.899345] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12735.357825] Lustre: server umount lustre-MDT0001 complete [12739.402935] Lustre: server umount lustre-OST0000 complete [12747.188620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12751.391738] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12753.876858] Lustre: 335760:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.146@tcp [12756.004079] Lustre: server umount MGS complete [12764.058879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12764.352946] LustreError: 336333:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12764.373041] LustreError: 336333:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12764.405758] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12768.339901] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12769.762795] LustreError: 336334:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12776.166037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12776.608266] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12780.752235] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12788.220652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12788.605803] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12794.553605] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12801.030749] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12802.388929] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12807.705590] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12809.123564] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12813.287117] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12813.289519] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12813.300932] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12813.314192] Lustre: Skipped 3 previous similar messages [12815.177849] Lustre: server umount lustre-OST0000 complete [12818.414957] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12818.439388] Lustre: Skipped 1 previous similar message [12818.446709] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12824.694225] Lustre: server umount lustre-MDT0000 complete [12827.997981] LustreError: 336315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784197679 with bad export cookie 17904980797215331308 [12827.999346] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12828.017272] LustreError: 336315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12828.317197] Lustre: server umount lustre-MDT0001 complete [12837.161725] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [12839.702329] Key type lgssc unregistered [12839.948281] LNet: 339767:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12839.955113] LNetError: 339767:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12840.998851] LNet: Removed LNI 192.168.204.146@tcp [12841.813257] Key type .llcrypt unregistered [12841.815949] Key type ._llcrypt unregistered [12857.079405] Key type ._llcrypt registered [12857.082039] Key type .llcrypt registered [12857.183826] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [12871.133869] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [12872.147883] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12872.168258] alg: No test for adler32 (adler32-zlib) [12873.187494] Lustre: Lustre: Build Version: 2.17.54_162_gdc75e25 [12873.446143] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [12875.136493] Key type lgssc registered [12876.159676] Lustre: Echo OBD driver; http://www.lustre.org/ [12909.332061] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12909.353200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12910.565046] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12910.607446] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12910.684809] Lustre: lustre-MDT0000: new disk, initializing [12910.772256] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12910.789929] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12914.818615] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12924.523324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12924.641823] Lustre: 342927:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12924.695704] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12924.699736] Lustre: Skipped 1 previous similar message [12924.754123] Lustre: lustre-MDT0001: new disk, initializing [12924.852796] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12924.904524] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12924.917474] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12928.726982] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12938.036810] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12943.376406] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12949.706911] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12949.934819] Lustre: lustre-OST0000: new disk, initializing [12949.937853] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12949.941650] Lustre: 344896:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12950.029890] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12955.231253] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12956.728219] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12956.737723] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12956.823137] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12963.710867] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12968.768463] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12968.953788] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12972.862575] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12973.026678] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12977.121312] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12977.136060] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12977.152767] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12977.641045] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12977.653194] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12977.682144] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12980.843650] Lustre: server umount lustre-OST0000 complete [12986.337196] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12986.342525] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12986.351525] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12990.100185] Lustre: server umount lustre-MDT0000 complete [12992.863583] LustreError: 342919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784197844 with bad export cookie 17289004927348504349 [12992.866961] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12992.875332] LustreError: 342919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12993.060933] Lustre: server umount lustre-MDT0001 complete [12999.838595] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 06:30:50 (1784197850) [13014.219263] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [13023.508158] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [13051.992426] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13052.169638] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13052.201343] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13052.260345] Lustre: lustre-MDT0000: new disk, initializing [13052.336835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13052.353082] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13055.781876] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [13064.195719] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13064.282751] Lustre: 349653:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13064.350767] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13064.355566] Lustre: Skipped 1 previous similar message [13064.407547] Lustre: lustre-MDT0001: new disk, initializing [13064.466402] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13064.490395] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13064.498890] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13068.299589] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [13076.825317] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13081.336503] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13086.778113] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13087.010319] Lustre: lustre-OST0000: new disk, initializing [13087.018265] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13087.034690] Lustre: 351621:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13087.134202] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13091.712591] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [13094.440621] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13094.448873] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13094.537082] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13099.559909] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13130.209608] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13130.220131] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13130.231613] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13130.231625] Lustre: Skipped 2 previous similar messages [13130.261620] Lustre: Skipped 2 previous similar messages [13132.817401] Lustre: server umount lustre-OST0000 complete [13136.356659] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13136.366982] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13136.391029] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13136.394643] Lustre: Skipped 1 previous similar message [13145.570761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13145.580214] Lustre: Skipped 3 previous similar messages [13150.176333] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13150.355725] Lustre: server umount lustre-MDT0000 complete [13153.599656] LustreError: 352488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784198005 with bad export cookie 17289004927348505231 [13153.606109] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13153.609750] LustreError: 352488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13153.899456] Lustre: server umount lustre-MDT0001 complete [13162.664804] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [13164.896752] Key type lgssc unregistered [13165.202623] LNet: 353774:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13165.208264] LNetError: 353774:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13165.227033] LNet: Removed LNI 192.168.204.146@tcp [13165.907314] Key type .llcrypt unregistered [13165.910970] Key type ._llcrypt unregistered [13233.874669] Key type ._llcrypt registered [13233.877877] Key type .llcrypt registered [13233.961804] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [13234.173643] Key type .llcrypt unregistered [13234.175769] Key type ._llcrypt unregistered [13295.190789] Key type ._llcrypt registered [13295.195336] Key type .llcrypt registered [13295.283446] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [13295.604933] Key type .llcrypt unregistered [13295.613668] Key type ._llcrypt unregistered [13320.612754] Key type ._llcrypt registered [13320.615865] Key type .llcrypt registered [13320.733312] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [13320.972577] Key type .llcrypt unregistered [13320.974744] Key type ._llcrypt unregistered [13344.897699] Key type ._llcrypt registered [13344.899504] Key type .llcrypt registered [13344.987442] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [13345.202654] Key type .llcrypt unregistered [13345.206906] Key type ._llcrypt unregistered [13421.630366] Key type ._llcrypt registered [13421.632015] Key type .llcrypt registered [13421.707156] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [13421.919934] Key type .llcrypt unregistered [13421.927618] Key type ._llcrypt unregistered [13444.896441] Key type ._llcrypt registered [13444.898342] Key type .llcrypt registered [13445.018432] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [13445.233325] Key type .llcrypt unregistered [13445.239234] Key type ._llcrypt unregistered [13467.810255] Key type ._llcrypt registered [13467.813570] Key type .llcrypt registered [13467.915291] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [13468.150470] Key type .llcrypt unregistered [13468.153410] Key type ._llcrypt unregistered [13490.384731] Key type ._llcrypt registered [13490.386402] Key type .llcrypt registered [13490.463367] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [13490.693262] Key type .llcrypt unregistered [13490.695635] Key type ._llcrypt unregistered [13518.456594] Key type ._llcrypt registered [13518.461065] Key type .llcrypt registered [13518.601040] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [13518.950538] Key type .llcrypt unregistered [13518.953599] Key type ._llcrypt unregistered [13543.897962] Key type ._llcrypt registered [13543.902287] Key type .llcrypt registered [13543.997181] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [13544.238405] Key type .llcrypt unregistered [13544.242071] Key type ._llcrypt unregistered [13595.704776] Key type ._llcrypt registered [13595.706803] Key type .llcrypt registered [13595.853457] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [13596.143336] Key type .llcrypt unregistered [13596.147092] Key type ._llcrypt unregistered [13620.585282] Key type ._llcrypt registered [13620.587841] Key type .llcrypt registered [13620.721700] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [13620.994104] Key type .llcrypt unregistered [13620.999352] Key type ._llcrypt unregistered [13645.297445] Key type ._llcrypt registered [13645.299539] Key type .llcrypt registered [13645.385860] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [13645.639631] Key type .llcrypt unregistered [13645.641372] Key type ._llcrypt unregistered [13685.487951] Key type ._llcrypt registered [13685.490943] Key type .llcrypt registered [13685.598688] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [13696.697585] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [13697.336806] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13697.447572] alg: No test for adler32 (adler32-zlib) [13698.423991] Lustre: Lustre: Build Version: 2.17.54_162_gdc75e25 [13698.584775] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [13700.224198] Key type lgssc registered [13701.070540] Lustre: Echo OBD driver; http://www.lustre.org/ [13729.750952] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13729.780531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13730.988595] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13731.011769] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13731.060652] Lustre: tst250fs-MDT0000: new disk, initializing [13731.110410] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13731.122780] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13734.815283] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [13744.949213] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13745.165505] Lustre: 373744:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13745.275516] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13745.279923] Lustre: Skipped 1 previous similar message [13745.384781] Lustre: tst250fs-MDT0001: new disk, initializing [13745.516596] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13745.568290] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13745.591177] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13750.744996] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [13756.232953] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13764.159523] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13764.466516] Lustre: tst250fs-OST0000: new disk, initializing [13764.474861] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13764.483065] Lustre: 375376:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13764.569671] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13768.252249] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13768.267668] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13768.453154] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13770.668749] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [13779.520729] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13779.668280] Lustre: tst250fs-OST0001: new disk, initializing [13779.671842] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13779.678477] Lustre: 376247:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13779.771258] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13784.112992] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [13788.453994] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13789.389122] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13789.394799] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13789.493609] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13805.029399] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13805.034833] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13805.045568] Lustre: Skipped 3 previous similar messages [13805.053755] Lustre: Skipped 3 previous similar messages [13810.149100] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13815.272294] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13815.285482] Lustre: Skipped 6 previous similar messages [13819.360206] Lustre: tst250fs-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13819.512992] Lustre: server umount tst250fs-MDT0000 complete [13822.234333] LustreError: 373734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784198674 with bad export cookie 1575055531607516166 [13822.247940] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13822.468967] Lustre: server umount tst250fs-MDT0001 complete [13835.821380] Lustre: server umount tst250fs-OST0000 complete [13849.405538] Lustre: server umount tst250fs-OST0001 complete [13863.684788] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [13869.827654] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [13894.561281] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 06:45:44 (1784198744) [13895.588917] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13908.164871] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [13915.770898] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [13940.779867] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13674 sec ========= 06:46:30 (1784198790) [13942.147395] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 06:46:32 (1784198792) === [13945.180096] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 06:46:35 (1784198795) === [13976.395245] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [13978.461458] Key type lgssc unregistered [13978.685212] LNet: 385389:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13978.693165] LNetError: 385389:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13978.713381] LNet: Removed LNI 192.168.204.146@tcp [13979.337176] Key type .llcrypt unregistered [13979.342156] Key type ._llcrypt unregistered