[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 476098015 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002365] x2apic enabled [ 0.003010] Switched APIC routing to physical x2apic. [ 0.004012] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008011] pid_max: default: 32768 minimum: 301 [ 0.009129] LSM: Security Framework initializing [ 0.010065] Yama: becoming mindful. [ 0.011049] SELinux: Initializing. [ 0.012078] *** VALIDATE selinux *** [ 0.020615] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025517] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026149] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027111] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028111] *** VALIDATE tmpfs *** [ 0.030301] *** VALIDATE proc *** [ 0.031226] *** VALIDATE cgroup *** [ 0.032009] *** VALIDATE cgroup2 *** [ 0.034183] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035227] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037029] Spectre V2 : User space: Vulnerable [ 0.038009] Speculative Store Bypass: Vulnerable [ 0.041525] debug: unmapping init [mem 0xffffffff98059000-0xffffffff98060fff] [ 0.044194] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045730] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046026] ... version: 2 [ 0.047016] ... bit width: 48 [ 0.048014] ... generic registers: 4 [ 0.049017] ... value mask: 0000ffffffffffff [ 0.050015] ... max period: 00007fffffffffff [ 0.051018] ... fixed-purpose events: 3 [ 0.052012] ... event mask: 000000070000000f [ 0.053322] rcu: Hierarchical SRCU implementation. [ 0.055411] smp: Bringing up secondary CPUs ... [ 0.056612] x86: Booting SMP configuration: [ 0.057028] .... node #0, CPUs: #1 #2 #3 [ 0.061047] smp: Brought up 1 node, 4 CPUs [ 0.063014] smpboot: Max logical packages: 1 [ 0.064013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.227597] node 0 deferred pages initialised in 158ms [ 0.231115] devtmpfs: initialized [ 0.232270] x86/mm: Memory block size: 128MB [ 0.235000] gcov: version magic: 0x41383552 [ 0.237313] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.238085] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.239263] pinctrl core: initialized pinctrl subsystem [ 0.240193] [ 0.240575] ************************************************************* [ 0.241017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.242018] ** ** [ 0.243014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.244014] ** ** [ 0.245012] ** This means that this kernel is built to expose internal ** [ 0.246015] ** IOMMU data structures, which may compromise security on ** [ 0.247012] ** your system. ** [ 0.248011] ** ** [ 0.249011] ** If you see this message and you are not debugging the ** [ 0.250012] ** kernel, report this immediately to your vendor! ** [ 0.251012] ** ** [ 0.252014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.253012] ************************************************************* [ 0.254522] NET: Registered protocol family 16 [ 0.256630] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.260082] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.263096] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.267148] cpuidle: using governor menu [ 0.270544] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.273695] PCI: Using configuration type 1 for base access [ 0.276162] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.288143] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.291048] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.295135] cryptd: max_cpu_qlen set to 1000 [ 0.299349] ACPI: Added _OSI(Module Device) [ 0.301021] ACPI: Added _OSI(Processor Device) [ 0.303018] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.305019] ACPI: Added _OSI(Processor Aggregator Device) [ 0.310253] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.317709] ACPI: Interpreter enabled [ 0.320077] ACPI: PM: (supports S0 S3 S4 S5) [ 0.322019] ACPI: Using IOAPIC for interrupt routing [ 0.324158] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.328409] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.339542] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.342058] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.345026] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.349112] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.354265] acpiphp: Slot [2] registered [ 0.356339] acpiphp: Slot [5] registered [ 0.357218] acpiphp: Slot [6] registered [ 0.358216] acpiphp: Slot [7] registered [ 0.360132] acpiphp: Slot [8] registered [ 0.362166] acpiphp: Slot [9] registered [ 0.364162] acpiphp: Slot [10] registered [ 0.365223] acpiphp: Slot [3] registered [ 0.367142] acpiphp: Slot [4] registered [ 0.369201] acpiphp: Slot [11] registered [ 0.372171] acpiphp: Slot [12] registered [ 0.373152] acpiphp: Slot [13] registered [ 0.375164] acpiphp: Slot [14] registered [ 0.377149] acpiphp: Slot [15] registered [ 0.379117] acpiphp: Slot [16] registered [ 0.381158] acpiphp: Slot [17] registered [ 0.383192] acpiphp: Slot [18] registered [ 0.385154] acpiphp: Slot [19] registered [ 0.386172] acpiphp: Slot [20] registered [ 0.388157] acpiphp: Slot [21] registered [ 0.390121] acpiphp: Slot [22] registered [ 0.391232] acpiphp: Slot [23] registered [ 0.392152] acpiphp: Slot [24] registered [ 0.394169] acpiphp: Slot [25] registered [ 0.396153] acpiphp: Slot [26] registered [ 0.398181] acpiphp: Slot [27] registered [ 0.400119] acpiphp: Slot [28] registered [ 0.401196] acpiphp: Slot [29] registered [ 0.403146] acpiphp: Slot [30] registered [ 0.405165] acpiphp: Slot [31] registered [ 0.407080] PCI host bridge to bus 0000:00 [ 0.409031] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.411037] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.414030] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.417034] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.419032] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.423037] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.425208] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.429163] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.432073] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.440683] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.446138] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.449025] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.451026] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.454027] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.458523] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.461848] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.465044] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.467905] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.474021] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.487036] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.493940] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.499570] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.508018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.515021] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.533020] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.544941] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.553023] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.560021] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.583022] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.594033] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.601021] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.607023] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.620050] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.632158] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.639019] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.646020] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.666023] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.676890] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.682016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.688017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.703032] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.713516] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.722021] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.732018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.750021] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.761377] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.765344] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.767327] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.769331] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.772271] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.775847] iommu: Default domain type: Passthrough [ 0.780362] SCSI subsystem initialized [ 0.782141] ACPI: bus type USB registered [ 0.783135] usbcore: registered new interface driver usbfs [ 0.785103] usbcore: registered new interface driver hub [ 0.788119] usbcore: registered new device driver usb [ 0.790189] pps_core: LinuxPPS API ver. 1 registered [ 0.792016] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.795084] PTP clock support registered [ 0.797238] EDAC MC: Ver: 3.0.0 [ 0.800321] PCI: Using ACPI for IRQ routing [ 0.802257] NetLabel: Initializing [ 0.803014] NetLabel: domain hash size = 128 [ 0.804016] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.805122] NetLabel: unlabeled traffic allowed by default [ 0.808150] vgaarb: loaded [ 0.810439] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.812023] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.818671] clocksource: Switched to clocksource kvm-clock [ 0.930717] VFS: Disk quotas dquot_6.6.0 [ 0.932643] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.936697] *** VALIDATE ramfs *** [ 0.938183] *** VALIDATE hugetlbfs *** [ 0.940148] pnp: PnP ACPI init [ 0.942858] pnp: PnP ACPI: found 6 devices [ 0.959743] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.963322] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.966228] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.969023] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.972051] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.975098] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.979669] NET: Registered protocol family 2 [ 0.982914] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.988683] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.993255] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.999403] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.003146] TCP: Hash tables configured (established 65536 bind 65536) [ 1.006333] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.009839] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.013180] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.016306] NET: Registered protocol family 1 [ 1.019889] RPC: Registered named UNIX socket transport module. [ 1.022204] RPC: Registered udp transport module. [ 1.024087] RPC: Registered tcp transport module. [ 1.025616] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.028181] NET: Registered protocol family 44 [ 1.030174] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.032793] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.035417] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.038238] PCI: CLS 0 bytes, default 64 [ 1.040327] Unpacking initramfs... [ 2.494720] debug: unmapping init [mem 0xffff9debfcc54000-0xffff9debfffbffff] [ 2.502890] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.505746] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.510629] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.002968] Initialise system trusted keyrings [ 3.006157] Key type blacklist registered [ 3.016712] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.030136] zbud: loaded [ 3.034081] *** VALIDATE nfs *** [ 3.035562] *** VALIDATE nfs4 *** [ 3.037544] pstore: using deflate compression [ 3.041479] Platform Keyring initialized [ 3.137647] NET: Registered protocol family 38 [ 3.139324] Key type asymmetric registered [ 3.141073] Asymmetric key parser 'x509' registered [ 3.143222] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.147106] io scheduler mq-deadline registered [ 3.148538] io scheduler kyber registered [ 3.150330] io scheduler bfq registered [ 3.152593] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.156160] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.158900] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.162259] ACPI: Power Button [PWRF] [ 3.169036] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.176267] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.189048] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.201086] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.221236] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.249645] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.278670] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.284814] Non-volatile memory driver v1.3 [ 3.286477] Linux agpgart interface v0.103 [ 3.320224] virtio_blk virtio1: [vda] 146712 512-byte logical blocks (75.1 MB/71.6 MiB) [ 3.324782] vda: detected capacity change from 0 to 75116544 [ 3.344027] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.347522] vdb: detected capacity change from 0 to 1073741824 [ 3.370101] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.374278] vdc: detected capacity change from 0 to 2621440000 [ 3.401119] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.404135] vdd: detected capacity change from 0 to 2621440000 [ 3.424243] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.427899] vde: detected capacity change from 0 to 4294967296 [ 3.443548] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.446686] vdf: detected capacity change from 0 to 4294967296 [ 3.453417] libphy: Fixed MDIO Bus: probed [ 3.460341] usbcore: registered new interface driver usbserial_generic [ 3.463071] usbserial: USB Serial support registered for generic [ 3.465611] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.468772] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.470987] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.473497] mousedev: PS/2 mouse device common for all mice [ 3.476700] rtc_cmos 00:05: RTC can wake from S4 [ 3.479817] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.482650] rtc_cmos 00:05: registered as rtc0 [ 3.487358] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.490649] intel_pstate: CPU model not supported [ 3.493750] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.498513] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.498713] hid: raw HID events driver (C) Jiri Kosina [ 3.504299] usbcore: registered new interface driver usbhid [ 3.506364] usbhid: USB HID core driver [ 3.508268] drop_monitor: Initializing network drop monitor service [ 3.510890] Initializing XFRM netlink socket [ 3.513049] NET: Registered protocol family 10 [ 3.515717] Segment Routing with IPv6 [ 3.517176] NET: Registered protocol family 17 [ 3.518860] mpls_gso: MPLS GSO support [ 3.523817] RAS: Correctable Errors collector initialized. [ 3.525480] AVX version of gcm_enc/dec engaged. [ 3.526865] AES CTR mode by8 optimization enabled [ 3.602895] sched_clock: Marking stable (3602840297, 0)->(4555064017, -952223720) [ 3.607253] registered taskstats version 1 [ 3.609712] Loading compiled-in X.509 certificates [ 3.612219] zswap: loaded using pool lzo/zbud [ 3.645320] Key type big_key registered [ 3.660709] Key type encrypted registered [ 3.662365] ima: No TPM chip found, activating TPM-bypass! [ 3.664663] ima: Allocated hash algorithm: sha1 [ 3.666395] ima: No architecture policies found [ 3.668333] evm: Initialising EVM extended attributes: [ 3.670410] evm: security.selinux [ 3.671816] evm: security.ima [ 3.672980] evm: security.capability [ 3.674462] evm: HMAC attrs: 0x1 [ 3.677113] rtc_cmos 00:05: setting system clock to 2026-09-08 05:05:21 UTC (1788843921) [ 3.683746] debug: unmapping init [mem 0xffffffff99003000-0xffffffff991fffff] [ 3.687144] debug: unmapping init [mem 0xffffffff97d82000-0xffffffff98058fff] [ 3.695132] Write protecting the kernel read-only data: 28672k [ 3.698939] debug: unmapping init [mem 0xffffffff96403000-0xffffffff965fffff] [ 3.701969] debug: unmapping init [mem 0xffffffff96d14000-0xffffffff96dfffff] [ 3.740524] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.749915] systemd[1]: Detected virtualization kvm. [ 3.752297] systemd[1]: Detected architecture x86-64. [ 3.754447] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.785345] systemd[1]: No hostname configured. [ 3.787537] systemd[1]: Set hostname to . [ 3.789982] random: systemd: uninitialized urandom read (16 bytes read) [ 3.793417] systemd[1]: Initializing machine ID from random generator. [ 3.855541] random: ln: uninitialized urandom read (6 bytes read) [ 3.947053] random: systemd: uninitialized urandom read (16 bytes read) [ 3.950383] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.955605] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.959837] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local File Systems. [ OK ] Reached target Slices. [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Initrd Root Device. [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.615001] device-mapper: uevent: version 1.0.3 [ 4.617162] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.365680] virtio_net virtio0 ens2: renamed from eth0 [ 5.371201] random: fast init done [ 5.427262] scsi host0: ata_piix [ 5.450647] scsi host1: ata_piix [ 5.453084] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.457058] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.803542] dracut-initqueue[585]: RTNETLINK answers: File exists [ 10.179803] random: crng init done [ 10.181462] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.556410] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.651368] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.899635] SELinux: Disabled at runtime. [ 11.959244] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.968241] systemd[1]: Detected virtualization kvm. [ 11.970663] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.483165] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.486938] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.492456] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.498308] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.501616] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.508944] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.513149] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Control Socket. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Kernel Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target rpc_pipefs.target. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ 12.580278] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting udev Coldplug all Devices... Mounting POSIX Message Queue File System... [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Huge Pages File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Paths. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target RPC Port Mapper. Mounting Kernel Debug File System... [ OK ] Created slice system-getty.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.952340] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.261322] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.296063] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.422638] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.447849] EDAC sbridge: Ver: 1.1.2 [ 15.066275] Key type dns_resolver registered [ 15.372880] NFS: Registering the id_resolver key type [ 15.375812] Key type id_resolver registered [ 15.378799] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started Login Service. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg339-server login: [ 45.098225] hrtimer: interrupt took 1868921 ns [ 71.702121] libcfs: loading out-of-tree module taints kernel. [ 71.793747] Key type ._llcrypt registered [ 71.796555] Key type .llcrypt registered [ 72.004724] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 92.503685] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 94.185029] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 94.209339] alg: No test for adler32 (adler32-zlib) [ 95.723504] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 96.670652] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 98.415209] Key type lgssc registered [ 100.341841] Lustre: Echo OBD driver; http://www.lustre.org/ [ 118.412606] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 165.268389] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 178.162995] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 178.202533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 179.512157] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 179.555349] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 179.657367] Lustre: lustre-MDT0000: new disk, initializing [ 179.761989] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 179.779656] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 184.552288] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 197.914730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 198.022518] Lustre: 6521:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 198.066566] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 198.070762] Lustre: Skipped 1 previous similar message [ 198.185134] Lustre: lustre-MDT0001: new disk, initializing [ 198.238790] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 198.265637] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 198.276590] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 202.731509] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 207.277155] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 217.101715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 217.469674] Lustre: lustre-OST0000: new disk, initializing [ 217.478973] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 217.488638] Lustre: 8459:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 217.614554] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 217.737814] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 217.747941] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 217.833574] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 223.983547] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 240.816128] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 240.921946] Lustre: lustre-OST0001: new disk, initializing [ 240.924307] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 240.931094] Lustre: 9533:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 241.018226] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 247.525598] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 247.902799] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 247.912450] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 248.000160] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 260.855209] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 270.897884] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 277.767695] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing check_logdir /tmp/testlogs/ [ 284.018257] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing yml_node [ 288.121406] Lustre: DEBUG MARKER: Client: 2.17.57.111 [ 290.909090] Lustre: DEBUG MARKER: MDS: 2.17.57.111 [ 293.952714] Lustre: DEBUG MARKER: OSS: 2.17.57.111 [ 296.047796] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Sep 8 01:10:11 EDT 2026 [ 314.695454] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 316.284943] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 335.330160] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 335.332589] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 335.351199] Lustre: Skipped 1 previous similar message [ 335.362724] Lustre: Skipped 3 previous similar messages [ 338.398238] Lustre: server umount lustre-MDT0000 complete [ 341.761645] LustreError: 6514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788844259 with bad export cookie 13688465520426861781 [ 341.763046] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 341.777721] LustreError: 6514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 345.576672] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 345.584562] LustreError: 6529:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 345.585345] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 345.598852] Lustre: Skipped 3 previous similar messages [ 345.643164] LustreError: 6529:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 346.592686] LustreError: 6527:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 346.593805] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 346.626484] LustreError: 6527:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 346.652574] Lustre: Skipped 2 previous similar messages [ 351.712191] LustreError: 6528:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 351.712219] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 351.725064] LustreError: 6528:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 351.742926] Lustre: Skipped 1 previous similar message [ 356.320618] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 356.581710] Lustre: server umount lustre-MDT0001 complete [ 360.781569] Lustre: server umount lustre-OST0000 complete [ 365.314653] Lustre: server umount lustre-OST0001 complete [ 372.022855] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 381.641267] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 421.273232] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 421.523855] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 421.555492] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 421.640398] Lustre: lustre-MDT0000: new disk, initializing [ 421.717982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 421.730991] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 426.740148] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 437.383640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 437.454361] Lustre: 15091:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 437.503263] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 437.507347] Lustre: Skipped 1 previous similar message [ 437.568278] Lustre: lustre-MDT0001: new disk, initializing [ 437.641609] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 437.685247] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 437.701587] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 442.714145] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 452.564939] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 458.365249] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 465.136898] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 465.483384] Lustre: lustre-OST0000: new disk, initializing [ 465.492042] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 465.499402] Lustre: 17063:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 465.585142] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 467.405095] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 467.415664] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 467.585184] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 471.754790] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 481.508700] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 486.443959] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 486.641186] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 491.594026] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 491.916951] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 498.148716] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 498.150895] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 498.171181] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 498.172065] LustreError: Skipped 1 previous similar message [ 498.190444] Lustre: Skipped 1 previous similar message [ 500.573334] Lustre: server umount lustre-OST0000 complete [ 504.288435] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 504.302192] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 504.320369] Lustre: Skipped 1 previous similar message [ 508.384749] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 508.387218] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 508.398853] Lustre: Skipped 1 previous similar message [ 510.574679] Lustre: server umount lustre-MDT0000 complete [ 513.507341] LustreError: 15104:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 513.542627] LustreError: 15104:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 514.393376] LustreError: 15086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788844432 with bad export cookie 13688465520426863321 [ 514.395660] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 514.408059] LustreError: 15086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 514.712758] Lustre: server umount lustre-MDT0001 complete [ 520.215209] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 01:13:55 (1788844435) [ 542.160073] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 542.759874] LustreError: 19938:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 542.821220] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 547.185342] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 552.929542] LustreError: 19939:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 552.950328] LustreError: 19939:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 554.815350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 560.084223] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 567.719891] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 574.320103] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 576.238976] Lustre: Failing over lustre-MDT0000 [ 576.456238] Lustre: server umount lustre-MDT0000 complete [ 578.018397] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 578.020206] LustreError: 19933:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 578.030120] Lustre: Skipped 1 previous similar message [ 578.049432] LustreError: 19933:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 593.392429] Lustre: 3651:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788844495/real 1788844495] req@ffff9dec75067b80 x1875738900989824/t0(0) o400->MGC192.168.203.139@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788844511 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 593.431457] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 596.032494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 603.617121] LustreError: 3650:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9dec7f7e6d80 x1875738900992768/t0(0) o250->MGC192.168.203.139@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 603.925813] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 608.594444] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 609.255613] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 609.257264] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 609.301119] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 618.107124] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 623.592818] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 623.596160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 623.620065] Lustre: Skipped 1 previous similar message [ 623.644609] Lustre: Skipped 2 previous similar messages [ 636.383176] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 636.598924] Lustre: server umount lustre-MDT0000 complete [ 639.850149] LustreError: 19918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788844557 with bad export cookie 13688465520426864735 [ 639.851870] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 639.859745] LustreError: 19918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 640.158762] Lustre: server umount lustre-MDT0001 complete [ 649.444433] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 650.172374] LustreError: 23288:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 650.209133] LustreError: 23288:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [ 650.290578] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 650.294446] Lustre: Skipped 2 previous similar messages [ 655.263700] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 664.583726] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 669.736533] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 677.463516] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 682.860469] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 685.103910] Lustre: Failing over lustre-MDT0000 [ 685.305764] Lustre: server umount lustre-MDT0000 complete [ 685.538046] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 685.538835] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 704.081979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 704.200485] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 704.469497] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 709.209880] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 709.547019] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 709.562860] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 709.579111] Lustre: Skipped 1 previous similar message [ 709.595141] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 718.903993] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 724.980767] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 724.983738] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 725.028290] Lustre: Skipped 7 previous similar messages [ 729.355156] Lustre: server umount lustre-MDT0000 complete [ 730.082631] LustreError: 24530:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 730.105947] LustreError: 24530:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 733.026620] LustreError: 23270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788844650 with bad export cookie 13688465520426865673 [ 733.043951] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 733.046121] LustreError: 23270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 733.401423] Lustre: server umount lustre-MDT0001 complete [ 741.939789] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 01:17:37 (1788844657) [ 760.327080] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 772.164395] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 810.362461] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 819.737805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 819.908493] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 819.944830] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 820.002858] Lustre: lustre-MDT0000: new disk, initializing [ 820.076379] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 820.082828] Lustre: Skipped 2 previous similar messages [ 820.095355] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 823.738281] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 832.557264] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 832.658570] Lustre: 29661:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 832.971274] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 837.768878] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 848.831673] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 854.806230] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 860.680371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 861.061186] Lustre: 31629:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 862.454746] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 862.480494] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 862.522903] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 862.534896] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 866.928374] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 876.461660] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 882.031310] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 903.656085] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 903.680182] Lustre: Skipped 3 previous similar messages [ 903.683478] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 903.685107] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 903.698668] Lustre: Skipped 1 previous similar message [ 909.422733] Lustre: server umount lustre-OST0000 complete [ 919.516968] Lustre: server umount lustre-MDT0000 complete [ 923.328415] LustreError: 29653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788844841 with bad export cookie 13688465520426866079 [ 923.330466] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 923.351270] LustreError: 29653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 923.704927] Lustre: server umount lustre-MDT0001 complete [ 934.236914] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 937.161418] Key type lgssc unregistered [ 937.513120] LNet: 33607:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 937.519362] LNetError: 33607:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 937.536115] LNet: Removed LNI 192.168.203.139@tcp [ 938.516098] Key type .llcrypt unregistered [ 938.524814] Key type ._llcrypt unregistered [ 966.328670] Key type ._llcrypt registered [ 966.332410] Key type .llcrypt registered [ 966.420312] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 967.829061] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 967.937770] alg: No test for adler32 (adler32-zlib) [ 969.216427] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 969.596920] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 971.383840] Key type lgssc registered [ 972.910589] Lustre: Echo OBD driver; http://www.lustre.org/ [ 982.183578] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 982.228908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 983.719181] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 988.372660] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 993.260770] LustreError: 35017:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 993.292381] LustreError: 35017:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 997.789931] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 998.364401] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1003.381365] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1012.429839] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1018.253224] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1025.521437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1025.956429] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1029.079824] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1033.772742] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1042.410836] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1043.464048] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1058.764768] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 01:22:54 (1788844974) [ 1061.182707] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1063.002240] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1070.787119] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 01:23:06 (1788844986) [ 1081.384665] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 01:23:16 (1788844996) [ 1082.944767] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1093.869509] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1093.874951] Lustre: Skipped 2 previous similar messages [ 1096.570386] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 01:23:31 (1788845011) [ 1098.714454] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1098.719299] Lustre: Skipped 2 previous similar messages [ 1107.358820] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1107.364331] Lustre: Skipped 4 previous similar messages [ 1121.685295] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 01:23:57 (1788845037) [ 1124.933841] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1128.480907] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1136.589200] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 01:24:12 (1788845052) [ 1138.811678] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1138.819092] Lustre: Skipped 6 previous similar messages [ 1150.984357] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 01:24:26 (1788845066) [ 1166.313267] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1166.314309] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1166.354802] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1171.350866] Lustre: server umount lustre-OST0000 complete [ 1176.546271] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1176.554634] Lustre: Skipped 1 previous similar message [ 1176.558566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1176.561981] Lustre: Skipped 2 previous similar messages [ 1181.669446] LustreError: 35013:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1181.687344] LustreError: 35013:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1181.712243] Lustre: server umount lustre-MDT0000 complete [ 1185.995057] LustreError: 34999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788845103 with bad export cookie 17379319734900258981 [ 1185.998181] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1186.019316] LustreError: 34999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1186.408109] Lustre: server umount lustre-MDT0001 complete [ 1196.770713] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 1199.559655] Key type lgssc unregistered [ 1199.939086] LNet: 42888:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1199.945286] LNetError: 42888:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1199.957238] LNet: Removed LNI 192.168.203.139@tcp [ 1201.028084] Key type .llcrypt unregistered [ 1201.032887] Key type ._llcrypt unregistered [ 1218.082603] Key type ._llcrypt registered [ 1218.085841] Key type .llcrypt registered [ 1218.202880] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 1234.869596] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 1235.968704] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1236.002237] alg: No test for adler32 (adler32-zlib) [ 1237.136252] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 1237.411929] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 1239.103589] Key type lgssc registered [ 1240.280620] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1279.611466] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1279.671488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1281.053829] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1281.118283] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1281.284383] Lustre: lustre-MDT0000: new disk, initializing [ 1281.405592] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1281.442199] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1287.317371] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1298.737098] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1298.868514] Lustre: 46038:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1298.919506] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1298.928996] Lustre: Skipped 1 previous similar message [ 1299.005890] Lustre: lustre-MDT0001: new disk, initializing [ 1299.119217] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1299.189234] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1299.211255] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1304.733192] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1314.794757] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1321.642505] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1329.136788] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1329.456790] Lustre: lustre-OST0000: new disk, initializing [ 1329.461805] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1329.470353] Lustre: 48007:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1329.572096] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1335.604721] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1337.919737] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1337.925365] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1338.011291] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1344.280257] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1348.212200] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1348.420740] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1352.538718] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1352.733186] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1358.820193] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1358.825611] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1358.832380] LustreError: Skipped 1 previous similar message [ 1358.834859] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1358.869938] Lustre: Skipped 1 previous similar message [ 1360.542521] Lustre: server umount lustre-OST0000 complete [ 1365.985086] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1365.991067] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1366.010896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1366.019701] Lustre: Skipped 1 previous similar message [ 1369.058757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1369.061051] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1370.606532] Lustre: server umount lustre-MDT0000 complete [ 1374.275414] LustreError: 46031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788845292 with bad export cookie 15492700558781581429 [ 1374.278299] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1374.287226] LustreError: 46031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1374.571509] Lustre: server umount lustre-MDT0001 complete [ 1383.558784] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 01:28:19 (1788845299) [ 1390.555958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1391.166914] LustreError: 50470:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1391.183989] LustreError: 50470:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1391.268559] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1394.765548] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1396.711301] LustreError: 50471:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1401.825799] LustreError: 50470:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1402.190675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1402.452501] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1407.001451] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1413.956496] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1419.696921] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1427.388669] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1427.811040] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1434.267608] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1441.971728] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1446.550665] LustreError: 52997:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1447.870808] Lustre: Modifying parameter general.timeout=1 in log params [ 1449.096866] Lustre: Modifying parameter general.timeout=2 in log params [ 1451.180928] Lustre: Modifying parameter general.timeout=4 in log params [ 1451.188377] Lustre: Skipped 1 previous similar message [ 1455.613310] Lustre: Modifying parameter general.timeout=8 in log params [ 1455.619063] Lustre: Skipped 3 previous similar messages [ 1463.822274] Lustre: Modifying parameter general.timeout=16 in log params [ 1463.826184] Lustre: Skipped 7 previous similar messages [ 1480.631586] Lustre: Modifying parameter general.timeout=33 in log params [ 1480.641193] Lustre: Skipped 16 previous similar messages [ 1513.148428] Lustre: Modifying parameter general.timeout=68 in log params [ 1513.155131] Lustre: Skipped 34 previous similar messages [ 1580.509306] Lustre: Disabling parameter general.timeout= in log params [ 1580.516459] Lustre: Skipped 61 previous similar messages [ 1584.464885] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 01:31:39 (1788845499) [ 1587.565805] Lustre: 59943:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1589.836609] Lustre: 60055:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1592.053481] Lustre: 60154:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1596.380987] Lustre: 60350:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=50 (mode = 2) failed: rc = -17 [ 1596.394857] Lustre: 60350:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1627.580765] Lustre: 61828:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1627.587978] Lustre: 61828:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1627.606809] LustreError: 61828:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1631.974404] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 01:32:27 (1788845547) [ 1648.096382] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1648.111500] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1648.129522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1648.132478] Lustre: Skipped 1 previous similar message [ 1651.769862] Lustre: server umount lustre-MDT0000 complete [ 1654.245668] LustreError: 50470:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1654.281597] LustreError: 50470:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1655.670661] LustreError: 50452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788845573 with bad export cookie 15492700558781582318 [ 1655.674216] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1655.681803] LustreError: 50452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1656.148495] Lustre: server umount lustre-MDT0001 complete [ 1671.128300] Lustre: server umount lustre-OST0000 complete [ 1701.889900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1702.440772] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1703.020454] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1709.346445] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1719.427535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1719.647924] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1719.687479] Lustre: 64360:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1724.819693] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1733.657566] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1741.349091] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1749.506554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1749.788803] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1750.246582] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1750.252021] Lustre: Skipped 1 previous similar message [ 1757.227448] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1765.706607] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1767.480274] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1767.486803] Lustre: Skipped 23 previous similar messages [ 1767.533150] Lustre: 66928:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1778.146656] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1778.152580] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1778.170975] LustreError: Skipped 1 previous similar message [ 1778.171850] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1778.189796] Lustre: Skipped 3 previous similar messages [ 1778.212182] Lustre: Skipped 4 previous similar messages [ 1780.895366] Lustre: server umount lustre-OST0000 complete [ 1786.851453] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1786.865061] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1786.889387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1791.585073] Lustre: server umount lustre-MDT0000 complete [ 1793.505075] LustreError: 64367:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1793.522930] LustreError: 64367:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1795.788283] LustreError: 66930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788845713 with bad export cookie 15492700558781585468 [ 1795.796155] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1795.803311] LustreError: 66930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1796.263298] Lustre: server umount lustre-MDT0001 complete [ 1807.912224] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 1810.374684] Key type lgssc unregistered [ 1810.717708] LNet: 68139:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1810.734411] LNetError: 68139:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1810.755693] LNet: Removed LNI 192.168.203.139@tcp [ 1811.595063] Key type .llcrypt unregistered [ 1811.596897] Key type ._llcrypt unregistered [ 1836.631468] Key type ._llcrypt registered [ 1836.634581] Key type .llcrypt registered [ 1836.780898] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 1838.141779] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1838.185927] alg: No test for adler32 (adler32-zlib) [ 1839.213696] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 1839.393373] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 1841.039234] Key type lgssc registered [ 1842.260971] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1849.439865] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1849.481520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1850.892585] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1854.872659] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1861.094960] LustreError: 69551:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1861.110564] LustreError: 69551:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1863.741065] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1864.601754] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1869.510728] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1877.581553] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1883.605246] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1891.193989] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1891.794965] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1898.052743] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1905.155177] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1916.613781] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1925.603530] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1925.618056] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1925.636585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1929.697267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1929.700716] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1929.726812] Lustre: Skipped 1 previous similar message [ 1931.544394] Lustre: server umount lustre-MDT0000 complete [ 1934.820814] LustreError: 69551:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1934.838094] LustreError: 69551:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1935.415618] LustreError: 69531:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788845853 with bad export cookie 5763695004346543980 [ 1935.427298] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1935.430552] LustreError: 69531:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1935.804232] Lustre: server umount lustre-MDT0001 complete [ 1949.857613] Lustre: server umount lustre-OST0000 complete [ 1977.980444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1978.526451] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1978.597763] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1978.919681] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1983.593744] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1993.346985] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1993.604276] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1993.679620] Lustre: 74424:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1994.582915] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2000.542604] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2008.107565] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2014.818331] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2020.913942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2021.149028] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2021.481345] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2027.234397] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2034.629680] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2036.608995] Lustre: 77002:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2038.441545] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2043.372098] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2043.390353] Lustre: Skipped 1 previous similar message [ 2043.396913] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2043.403048] Lustre: Skipped 3 previous similar messages [ 2048.487174] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2048.489229] Lustre: Skipped 1 previous similar message [ 2049.675566] Lustre: server umount lustre-OST0000 complete [ 2053.600105] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2053.604370] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2053.618210] Lustre: Skipped 1 previous similar message [ 2053.642485] Lustre: Skipped 1 previous similar message [ 2059.694929] Lustre: server umount lustre-MDT0000 complete [ 2062.973399] LustreError: 74418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788845980 with bad export cookie 5763695004346544827 [ 2062.979979] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2062.986754] LustreError: 74418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2063.328643] Lustre: server umount lustre-MDT0001 complete [ 2073.400486] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 2075.823035] Key type lgssc unregistered [ 2076.167201] LNet: 78209:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2076.179375] LNetError: 78209:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2076.207791] LNet: Removed LNI 192.168.203.139@tcp [ 2076.955551] Key type .llcrypt unregistered [ 2076.958387] Key type ._llcrypt unregistered [ 2101.283493] Key type ._llcrypt registered [ 2101.286806] Key type .llcrypt registered [ 2101.393911] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 2102.550717] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2102.716561] alg: No test for adler32 (adler32-zlib) [ 2103.724772] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 2104.021283] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 2105.727410] Key type lgssc registered [ 2106.872271] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2115.160125] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2115.183575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2116.763103] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2121.332799] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2126.308783] LustreError: 79618:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2126.333303] LustreError: 79618:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2130.295855] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2130.928064] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2135.541739] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2143.126484] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2148.436488] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2156.278320] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2157.187917] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2164.255408] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2173.372705] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2182.738206] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2183.812186] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2184.890718] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2187.710616] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2187.716642] Lustre: Skipped 2 previous similar messages [ 2191.889363] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2191.896334] Lustre: Skipped 3 previous similar messages [ 2200.502519] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2200.508517] Lustre: Skipped 8 previous similar messages [ 2216.707086] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2216.711992] Lustre: Skipped 17 previous similar messages [ 2248.944557] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2248.953211] Lustre: Skipped 32 previous similar messages [ 2282.965400] Lustre: 87719:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2285.295975] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 01:43:21 (1788846201) [ 2286.738910] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2288.605260] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 01:43:24 (1788846204) [ 2293.729189] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2293.733676] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2293.767393] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2298.660343] Lustre: server umount lustre-OST0000 complete [ 2303.968544] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2303.987318] Lustre: Skipped 1 previous similar message [ 2303.995922] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2303.998654] Lustre: Skipped 1 previous similar message [ 2308.820336] Lustre: server umount lustre-MDT0000 complete [ 2309.088375] LustreError: 79613:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2309.119071] LustreError: 79613:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2312.553365] LustreError: 79599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788846230 with bad export cookie 9314931159601434200 [ 2312.566182] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2312.573950] LustreError: 79599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2313.043232] Lustre: server umount lustre-MDT0001 complete [ 2324.777573] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 2327.939426] Key type lgssc unregistered [ 2328.247813] LNet: 88969:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2328.256865] LNetError: 88969:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2329.339912] LNet: Removed LNI 192.168.203.139@tcp [ 2330.501233] Key type .llcrypt unregistered [ 2330.504294] Key type ._llcrypt unregistered [ 2335.692964] Key type ._llcrypt registered [ 2335.695589] Key type .llcrypt registered [ 2335.807313] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2340.864493] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules [ 2342.258047] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2342.304699] alg: No test for adler32 (adler32-zlib) [ 2343.330667] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 2343.703640] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 2345.416502] Key type lgssc registered [ 2346.989439] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2348.411802] LustreError: 90174:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2367.585442] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 2371.087363] LustreError: 90174:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2378.386029] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2378.404522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2379.911139] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2384.781793] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2389.473029] LustreError: 91188:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2389.510321] LustreError: 91188:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2394.530417] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 01:45:10 (1788846310) [ 2394.626225] LustreError: 91187:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2399.723996] LustreError: 91188:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2399.910121] LustreError: 91219:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2399.933817] LustreError: 91219:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2399.945846] LustreError: 91219:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 20, retries 0, failed: rc = -5 [ 2400.558204] Lustre: server umount lustre-MDT0000 complete [ 2416.379527] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 2419.384233] Key type lgssc unregistered [ 2419.724941] LNet: 92711:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2419.740830] LNetError: 92711:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2420.777511] LNet: Removed LNI 192.168.203.139@tcp [ 2421.884663] Key type .llcrypt unregistered [ 2421.887964] Key type ._llcrypt unregistered [ 2439.530744] Key type ._llcrypt registered [ 2439.533434] Key type .llcrypt registered [ 2439.636847] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 2440.765113] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2440.992122] alg: No test for adler32 (adler32-zlib) [ 2442.081276] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 2442.374117] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 2444.039276] Key type lgssc registered [ 2445.558331] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2454.610412] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2454.627746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2456.156958] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2460.883759] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2465.262207] LustreError: 93925:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2465.303986] LustreError: 93925:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2470.375441] LustreError: 93926:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2471.874403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2472.321082] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2477.969443] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2490.908297] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2500.661822] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2509.953179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2510.851102] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2517.431519] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2525.417455] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2577.091177] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 01:48:12 (1788846492) [ 2579.245937] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2581.147900] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 01:48:16 (1788846496) [ 2586.593411] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2586.600274] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2586.614589] Lustre: Skipped 1 previous similar message [ 2586.639541] Lustre: Skipped 2 previous similar messages [ 2591.720786] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2592.565313] Lustre: server umount lustre-MDT0000 complete [ 2596.197685] LustreError: 93907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788846514 with bad export cookie 16784051755919661436 [ 2596.209930] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2596.219511] LustreError: 93907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2596.640834] Lustre: server umount lustre-MDT0001 complete [ 2611.049521] Lustre: server umount lustre-OST0000 complete [ 2624.197922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2624.589580] LustreError: 98222:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2624.804133] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2629.505957] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2630.114504] LustreError: 98223:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2634.210603] LustreError: 98222:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2638.205213] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2638.978510] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2644.111637] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2652.637321] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2659.075506] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2672.969139] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2673.285954] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2673.295964] LustreError: 98210:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2673.330293] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2673.340694] LustreError: 100305:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -98 [ 2673.354766] LustreError: 100305:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2673.360962] LustreError: 100305:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2673.485339] Lustre: server umount lustre-OST0000 complete [ 2673.491970] LustreError: 100305:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2683.388930] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2683.595711] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2683.602668] LustreError: 98210:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2683.639791] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2683.646602] LustreError: 100770:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -98 [ 2683.652419] LustreError: 100770:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2683.655857] LustreError: 100770:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2683.723785] Lustre: server umount lustre-OST0000 complete [ 2683.729570] LustreError: 100770:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2693.525634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2693.820197] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2693.823657] Lustre: Found index 0 for lustre-OST0000, updating log [ 2693.836810] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2694.006947] Lustre: lustre-OST0000: new disk, initializing [ 2694.011710] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2694.017742] Lustre: 101307:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2694.398987] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2702.219688] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2706.362153] Lustre: Failing over lustre-OST0000 [ 2706.790230] Lustre: server umount lustre-OST0000 complete [ 2713.569538] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2713.571592] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2713.581812] Lustre: Skipped 2 previous similar messages [ 2713.599443] Lustre: Skipped 3 previous similar messages [ 2718.692834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2718.705398] Lustre: Skipped 1 previous similar message [ 2720.811160] Lustre: server umount lustre-MDT0000 complete [ 2723.810028] LustreError: 98219:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2723.822324] LustreError: 98219:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2724.773868] LustreError: 98203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788846642 with bad export cookie 16784051755919664180 [ 2724.782546] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2724.797586] LustreError: 98203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2725.303371] Lustre: server umount lustre-MDT0001 complete [ 2736.176366] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 01:50:51 (1788846651) [ 2754.313409] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 2766.835309] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2767.101088] LustreError: 103654:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2767.195264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2772.480475] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2782.256960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2787.957099] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2791.299973] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2798.739184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2799.451112] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2799.455839] Lustre: Skipped 1 previous similar message [ 2799.475764] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2806.461432] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2807.658029] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2807.666402] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2807.684460] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2811.761824] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:05 [ 2811.774729] Lustre: lustre-OST0000: Recovery over after 0:04, of 1 clients 1 recovered and 0 were evicted. [ 2811.775131] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2811.783721] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2811.787228] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2816.356657] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2816.511966] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2816.610918] Lustre: lustre-OST0001: new disk, initializing [ 2816.615826] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2816.627334] Lustre: 105976:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2823.533458] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2824.359345] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2824.379260] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2824.571615] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2831.709525] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2840.075571] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2845.173096] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2845.176150] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2845.188463] Lustre: Skipped 2 previous similar messages [ 2845.205935] Lustre: Skipped 2 previous similar messages [ 2851.297339] Lustre: server umount lustre-MDT0001 complete [ 2852.289033] LustreError: 105986:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2852.324553] LustreError: 105986:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2862.635938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2862.834063] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2862.847425] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2862.903282] Lustre: 103644:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2863.818929] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2863.849924] Lustre: Skipped 1 previous similar message [ 2864.301818] LustreError: 107507:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2864.315650] LustreError: 107507:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2864.332817] LustreError: 103646:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2864.439450] LustreError: 107516:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.139@tcp: cfg command failed: rc = -17 [ 2864.459073] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2864.459073] [ 2869.137482] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2869.248388] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2879.955971] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 01:53:15 (1788846795) [ 2898.935624] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 2920.054897] Lustre: 109591:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2945.642658] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2949.683277] Lustre: 110727:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3037.663696] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3037.671216] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3037.701882] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3037.738119] Lustre: Skipped 4 previous similar messages [ 3040.218944] Lustre: server umount lustre-MDT0000 complete [ 3043.320961] LustreError: 105986:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3043.357879] LustreError: 105986:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 3044.762252] LustreError: 106697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788846962 with bad export cookie 16784051755919665181 [ 3044.766521] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3044.789020] LustreError: 106697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3045.325671] Lustre: server umount lustre-MDT0001 complete [ 3059.693703] Lustre: server umount lustre-OST0000 complete [ 3072.600498] Lustre: server umount lustre-OST0001 complete [ 3078.413701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3088.843237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3097.235335] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3107.528832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3129.123962] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 3141.240270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3141.274805] Lustre: lustre-MDT0000: reset Object Index mappings [ 3142.125700] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3147.285993] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3155.395467] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3155.411198] Lustre: lustre-MDT0001: reset Object Index mappings [ 3159.804460] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3162.738189] Lustre: 115826:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3169.830888] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3178.867228] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3181.923547] LustreError: 116182:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3181.934629] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:200 to 0x280000400:225) [ 3181.956777] LustreError: 116182:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3188.208219] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:381 to 0x280000401:418) [ 3188.857681] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3194.907985] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 3194.909046] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 3196.604167] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3204.175645] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3208.370930] Lustre: 117703:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3264.822544] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 01:59:40 (1788847180) [ 3292.130617] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3292.134655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3292.159995] Lustre: Skipped 5 previous similar messages [ 3292.177778] Lustre: Skipped 7 previous similar messages [ 3296.246378] Lustre: server umount lustre-MDT0000 complete [ 3299.842264] LustreError: 115828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788847217 with bad export cookie 16784051755919712256 [ 3299.847674] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3299.849225] LustreError: 115828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3302.381173] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3302.394732] Lustre: Skipped 2 previous similar messages [ 3306.426605] Lustre: server umount lustre-MDT0001 complete [ 3316.966499] Lustre: server umount lustre-OST0000 complete [ 3327.021924] Lustre: server umount lustre-OST0001 complete [ 3335.750643] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 3347.336035] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 3384.210797] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3384.474493] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3384.520507] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3384.627432] Lustre: lustre-MDT0000: new disk, initializing [ 3384.719493] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3389.177226] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3396.867199] Lustre: server umount lustre-MDT0000 complete [ 3407.129821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3407.367184] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3407.376979] Lustre: 121986:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3407.388546] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3407.623078] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3407.626264] Lustre: Skipped 4 previous similar messages [ 3412.551889] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3417.727381] Lustre: server umount lustre-MDT0000 complete [ 3426.258773] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 02:02:22 (1788847342) [ 3428.058654] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3429.948633] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 02:02:25 (1788847345) [ 3448.350922] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 3459.042405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3465.333652] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3474.391980] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3474.534801] Lustre: 124036:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3474.618783] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3474.624991] Lustre: Skipped 1 previous similar message [ 3474.747419] Lustre: lustre-MDT0001: new disk, initializing [ 3474.922202] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3474.933141] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3479.623617] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3484.733337] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3492.206239] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3492.573853] Lustre: lustre-OST0000: new disk, initializing [ 3492.577423] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3492.584575] Lustre: 125615:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3494.030038] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3494.050374] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3494.147119] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3498.924274] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3509.255051] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3509.429752] Lustre: lustre-OST0001: new disk, initializing [ 3509.434442] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3509.439581] Lustre: 126485:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3510.797213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3510.810846] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3510.950859] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3515.567201] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3525.477258] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3529.281713] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3938.748295] Lustre: 132900:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3938.772147] Lustre: 132900:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3938.785748] Lustre: 132900:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3938.795562] Lustre: 132900:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3938.806585] Lustre: 132900:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3938.821930] Lustre: 132900:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4293.601404] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 02:16:48 (1788848208) [ 4309.477637] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4309.501563] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4309.524535] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4309.528190] Lustre: Skipped 4 previous similar messages [ 4311.929208] Lustre: server umount lustre-MDT0000 complete [ 4315.130542] LustreError: 127291:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4315.152526] LustreError: 127291:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [ 4316.269947] LustreError: 124026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788848234 with bad export cookie 16784051755919782543 [ 4316.276396] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4316.295857] LustreError: 124026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4320.228497] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4320.240779] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4320.250435] Lustre: Skipped 4 previous similar messages [ 4320.269550] Lustre: Skipped 4 previous similar messages [ 4323.111056] Lustre: server umount lustre-MDT0001 complete [ 4333.876754] Lustre: server umount lustre-OST0000 complete [ 4344.369790] Lustre: server umount lustre-OST0001 complete [ 4352.196306] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 4364.878237] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 4410.314349] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4410.597924] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4410.629970] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4410.717281] Lustre: lustre-MDT0000: new disk, initializing [ 4410.824987] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4410.830682] Lustre: Skipped 4 previous similar messages [ 4410.845712] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4417.581581] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4432.625526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4432.808565] Lustre: 135965:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4432.889726] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4432.900054] Lustre: Skipped 1 previous similar message [ 4433.011294] Lustre: lustre-MDT0001: new disk, initializing [ 4433.166149] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4433.193943] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4438.068803] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4447.938922] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4454.620804] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4462.093589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4462.517434] Lustre: lustre-OST0000: new disk, initializing [ 4462.521613] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4462.530288] Lustre: 137932:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4463.673238] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4463.685093] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4463.830789] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4469.190347] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4479.433844] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4490.038188] Lustre: lustre-MDD0000: changelog on [ 4493.195256] Lustre: lustre-MDD0001: changelog on [ 6363.826435] Lustre: lustre-MDD0001: changelog off [ 6364.694113] Lustre: 135971:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6365.157692] Lustre: lustre-MDD0000: changelog off [ 6368.737532] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6368.738506] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6368.742525] LustreError: Skipped 1 previous similar message [ 6368.750442] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6368.754426] Lustre: Skipped 3 previous similar messages [ 6373.858805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6373.862316] Lustre: Skipped 1 previous similar message [ 6374.004297] Lustre: server umount lustre-OST0000 complete [ 6378.978803] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6378.978866] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6378.979724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6378.987153] Lustre: Skipped 2 previous similar messages [ 6379.000346] Lustre: Skipped 1 previous similar message [ 6384.097854] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6384.102342] Lustre: Skipped 1 previous similar message [ 6390.239171] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6390.374607] Lustre: server umount lustre-MDT0000 complete [ 6392.046435] LustreError: 135957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850309 with bad export cookie 16784051755920604798 [ 6392.050654] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6392.051778] LustreError: 135957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6392.249913] Lustre: server umount lustre-MDT0001 complete [ 6397.437519] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 6398.817968] Key type lgssc unregistered [ 6398.979143] LNet: 147096:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6398.983413] LNetError: 147096:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6398.995612] LNet: Removed LNI 192.168.203.139@tcp [ 6399.423138] Key type .llcrypt unregistered [ 6399.425031] Key type ._llcrypt unregistered [ 6409.420410] Key type ._llcrypt registered [ 6409.422172] Key type .llcrypt registered [ 6409.474455] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 6418.263557] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 6418.740771] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6418.807656] alg: No test for adler32 (adler32-zlib) [ 6419.707424] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 6419.815735] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 6421.431161] Key type lgssc registered [ 6421.903609] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6439.157239] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6439.166783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6440.284343] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6440.302517] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6440.354297] Lustre: lustre-MDT0000: new disk, initializing [ 6440.401805] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6440.412829] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6442.587566] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6448.207287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6448.269203] Lustre: 150265:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6448.300806] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6448.304658] Lustre: Skipped 1 previous similar message [ 6448.347619] Lustre: lustre-MDT0001: new disk, initializing [ 6448.400700] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6448.420431] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6448.426830] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6450.528137] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6455.827907] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6458.743178] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6462.349644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6462.481271] Lustre: lustre-OST0000: new disk, initializing [ 6462.483915] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6462.488079] Lustre: 152235:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6462.527555] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6465.067364] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6469.639654] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6470.146828] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6470.151634] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6470.193578] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6477.602914] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_module obdecho/obdecho [ 6478.382077] LustreError: 153241:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6478.388215] LustreError: 153241:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6479.134267] LustreError: 153337:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6484.451338] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6484.458740] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6484.471349] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6485.477357] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6485.477527] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6489.810342] Lustre: server umount lustre-MDT0000 complete [ 6491.932417] LustreError: 150258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850409 with bad export cookie 15244613514181895154 [ 6491.940503] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6491.941761] LustreError: 150258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6492.102704] Lustre: server umount lustre-MDT0001 complete [ 6503.349584] Lustre: server umount lustre-OST0000 complete [ 6513.125694] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 6517.768957] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 6533.632501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6533.735630] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6533.748995] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6533.791209] Lustre: lustre-MDT0000: new disk, initializing [ 6533.826092] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6533.836603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6535.505713] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6540.419178] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6540.470124] Lustre: 157043:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6540.503468] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6540.506770] Lustre: Skipped 1 previous similar message [ 6540.545461] Lustre: lustre-MDT0001: new disk, initializing [ 6540.586987] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6540.603431] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6540.610152] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6542.270324] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6546.520660] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6548.732407] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6551.611025] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6551.752577] Lustre: lustre-OST0000: new disk, initializing [ 6551.756746] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6551.762108] Lustre: 159012:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6551.812303] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6554.466631] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6558.211892] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6558.216017] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6558.257874] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6559.091202] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6561.117802] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6561.218776] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6562.978666] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6563.071835] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6568.416437] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6568.418719] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6568.423302] LustreError: Skipped 1 previous similar message [ 6568.424053] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6568.433510] Lustre: Skipped 2 previous similar messages [ 6568.445732] Lustre: Skipped 3 previous similar messages [ 6570.086793] Lustre: server umount lustre-OST0000 complete [ 6573.537637] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6573.538557] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6573.543882] Lustre: Skipped 1 previous similar message [ 6578.656697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6578.659703] Lustre: Skipped 2 previous similar messages [ 6586.335164] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6586.415165] Lustre: server umount lustre-MDT0000 complete [ 6587.994039] LustreError: 160552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850505 with bad export cookie 15244613514181896099 [ 6587.995184] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6588.001148] LustreError: 160552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6588.140796] Lustre: server umount lustre-MDT0001 complete [ 6589.709200] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 02:55:06 (1788850506) [ 6590.509091] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6591.301745] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 02:55:07 (1788850507) [ 6594.697805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6594.898164] LustreError: 161423:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6594.909080] LustreError: 161423:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 6594.929375] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6596.629403] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6600.161545] LustreError: 161424:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6600.240182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6602.100110] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6605.238865] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6607.636213] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6610.683421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6613.664816] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6617.272413] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6631.393059] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6631.400965] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6631.410760] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6631.413314] Lustre: Skipped 2 previous similar messages [ 6634.096309] Lustre: server umount lustre-MDT0000 complete [ 6635.488499] LustreError: 161423:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6635.495184] LustreError: 161423:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6635.749373] LustreError: 161449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850553 with bad export cookie 15244613514181896967 [ 6635.752427] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6635.754452] LustreError: 161449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6637.997675] LustreError: 161418:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6640.610440] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6640.617088] Lustre: Skipped 2 previous similar messages [ 6643.117253] LustreError: 161420:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6643.125367] LustreError: 161420:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6648.240095] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.39@tcp (stopping) [ 6648.245075] Lustre: Skipped 5 previous similar messages [ 6650.335155] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6650.444938] Lustre: server umount lustre-MDT0001 complete [ 6653.299699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6658.993280] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6659.132529] LustreError: 165279:0:(llog.c:1655:llog_backup()) MGC192.168.203.139@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6659.138777] Lustre: 165279:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.139@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6670.304522] LustreError: 165284:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6670.313342] LustreError: 165284:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6689.196807] LustreError: 165286:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6689.205240] LustreError: 165286:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 6720.479264] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6720.482290] Lustre: 165279:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.139@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6721.505193] LustreError: 165285:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6721.513540] LustreError: 165285:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 23 previous similar messages [ 6761.452643] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6761.459095] Lustre: Skipped 2 previous similar messages [ 6765.371870] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6766.565820] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6774.439119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6780.895844] LustreError: 148490:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000002925bd0b x1875745531078144/t0(0) o250->MGC192.168.203.139@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6781.060210] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -11 [ 6784.765581] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6786.546550] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6794.317880] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6796.698586] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6804.411412] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6806.351895] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6827.488637] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6827.490280] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6827.496413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6827.510767] Lustre: Skipped 2 previous similar messages [ 6832.075485] Lustre: server umount lustre-MDT0000 complete [ 6832.616492] LustreError: 165318:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6832.649410] LustreError: 165318:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 45 previous similar messages [ 6834.944391] LustreError: 161405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850752 with bad export cookie 15244613514181903407 [ 6834.945656] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6834.960786] LustreError: 161405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6835.154684] Lustre: server umount lustre-MDT0001 complete [ 6847.408760] Lustre: server umount lustre-OST0000 complete [ 6861.281552] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 6868.263099] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 6893.288274] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6893.443862] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6893.484809] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6893.568850] Lustre: lustre-MDT0000: new disk, initializing [ 6893.636964] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6893.642479] Lustre: Skipped 1 previous similar message [ 6893.664268] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6896.412521] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6904.287458] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6904.370325] Lustre: 170384:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6904.565361] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6908.456943] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6924.387539] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6934.208100] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6947.876094] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6948.247412] Lustre: lustre-OST0000: new disk, initializing [ 6948.250500] Lustre: Skipped 1 previous similar message [ 6948.254712] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6948.259866] Lustre: Skipped 2 previous similar messages [ 6948.264508] Lustre: 172366:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6957.254634] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6957.281938] Lustre: Skipped 1 previous similar message [ 6957.294381] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6957.521709] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6959.509975] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6971.122048] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6975.624507] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6975.860707] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6979.927200] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6980.147841] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6983.142504] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6983.156193] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6983.193684] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6983.199788] Lustre: Skipped 3 previous similar messages [ 6988.186775] Lustre: server umount lustre-OST0000 complete [ 7006.175155] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7006.339620] Lustre: server umount lustre-MDT0000 complete [ 7008.744927] LustreError: 170394:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7010.105702] LustreError: 171179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850927 with bad export cookie 15244613514181920242 [ 7010.110430] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7010.130043] LustreError: 171179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7010.653053] Lustre: server umount lustre-MDT0001 complete [ 7019.711510] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 03:02:15 (1788850935) [ 7026.075109] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7026.628323] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7026.636101] Lustre: Skipped 2 previous similar messages [ 7030.764275] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7040.252645] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7045.127533] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7052.571440] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7058.660240] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7065.652896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7072.716630] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7080.251557] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7093.578774] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 03:03:29 (1788851009) [ 7097.824802] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7097.838281] LustreError: Skipped 2 previous similar messages [ 7097.842959] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7097.863902] Lustre: Skipped 3 previous similar messages [ 7103.811078] Lustre: server umount lustre-OST0000 complete [ 7114.483067] Lustre: server umount lustre-MDT0000 complete [ 7118.474765] LustreError: 174757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851036 with bad export cookie 15244613514181921124 [ 7118.486019] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7118.495921] LustreError: 174757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7118.920766] Lustre: server umount lustre-MDT0001 complete [ 7130.862551] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 7133.998346] Key type lgssc unregistered [ 7134.359181] LNet: 178509:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7134.370585] LNetError: 178509:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7134.411291] LNet: Removed LNI 192.168.203.139@tcp [ 7135.352744] Key type .llcrypt unregistered [ 7135.356465] Key type ._llcrypt unregistered [ 7155.630936] Key type ._llcrypt registered [ 7155.633797] Key type .llcrypt registered [ 7155.757492] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 7156.967882] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7157.000377] alg: No test for adler32 (adler32-zlib) [ 7158.022597] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 7158.240845] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 7159.919229] Key type lgssc registered [ 7161.254601] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7170.068827] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7170.103693] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7185.759477] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7190.879917] LustreError: 179747:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.139@tcp: failed processing log, type 4: rc = -110 [ 7195.999995] LustreError: 179747:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7196.026063] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7196.040953] LustreError: MGC192.168.203.139@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7196.067935] LustreError: 179747:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7196.098362] LustreError: 179747:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -22 [ 7196.113204] LustreError: 179747:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 7196.222337] Lustre: server umount lustre-OST0000 complete [ 7196.234194] LustreError: 179747:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7206.099057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7207.747409] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7212.691228] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7218.156083] LustreError: 180246:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7218.186152] LustreError: 180246:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7222.210086] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7222.556587] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7228.400833] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7237.352490] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7243.764930] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7251.086184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7251.309467] LustreError: 182062:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7251.326222] Lustre: 182062:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.203.139@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7251.579136] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7259.195816] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7267.581757] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7273.457396] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7273.470809] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7273.486534] Lustre: Skipped 2 previous similar messages [ 7273.500243] Lustre: Skipped 2 previous similar messages [ 7276.538404] Lustre: server umount lustre-MDT0000 complete [ 7278.568827] LustreError: 180973:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7278.586482] LustreError: 180973:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7280.279140] LustreError: 180227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851198 with bad export cookie 15211748824850832845 [ 7280.300940] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7280.615922] Lustre: server umount lustre-MDT0001 complete [ 7294.951903] Lustre: server umount lustre-OST0000 complete [ 7305.467641] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 7317.154135] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 7356.453712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7356.740323] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7356.774262] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7356.859065] Lustre: lustre-MDT0000: new disk, initializing [ 7356.985608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7357.002034] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7361.215571] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7372.176676] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7372.293320] Lustre: 185589:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7372.388789] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7372.404086] Lustre: Skipped 1 previous similar message [ 7372.528884] Lustre: lustre-MDT0001: new disk, initializing [ 7372.727612] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7372.792110] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7372.804559] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7376.620255] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7386.811324] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7392.623826] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7399.336790] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7399.623745] Lustre: lustre-OST0000: new disk, initializing [ 7399.628390] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7399.641845] Lustre: 187555:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7399.795370] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7406.253954] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7407.680161] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7407.696326] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7407.822813] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7416.144216] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7421.793978] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7422.123820] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7427.387974] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7427.648252] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7433.698639] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7433.701053] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7433.702824] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7433.772112] Lustre: Skipped 1 previous similar message [ 7436.234095] Lustre: server umount lustre-OST0000 complete [ 7443.943879] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7443.949251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7443.954684] Lustre: Skipped 1 previous similar message [ 7443.970144] Lustre: Skipped 2 previous similar messages [ 7446.390456] Lustre: server umount lustre-MDT0000 complete [ 7449.057639] LustreError: 185599:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7449.072529] LustreError: 185599:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7449.852203] LustreError: 185579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851367 with bad export cookie 15211748824850833566 [ 7449.865436] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7449.869766] LustreError: 185579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 7450.164895] Lustre: server umount lustre-MDT0001 complete [ 7458.236075] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 03:09:34 (1788851374) [ 7475.384405] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 7477.788673] Key type lgssc unregistered [ 7478.033647] LNet: 190502:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7478.044570] LNetError: 190502:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7478.065827] LNet: Removed LNI 192.168.203.139@tcp [ 7478.869174] Key type .llcrypt unregistered [ 7478.872621] Key type ._llcrypt unregistered [ 7498.859143] Key type ._llcrypt registered [ 7498.867622] Key type .llcrypt registered [ 7498.941659] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 7500.503453] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7500.563900] alg: No test for adler32 (adler32-zlib) [ 7501.858422] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 7502.258508] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 7503.991244] Key type lgssc registered [ 7505.593264] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7513.803357] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7513.826510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7515.165245] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7519.096718] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7524.327835] LustreError: 191720:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7524.358126] LustreError: 191720:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7526.136486] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7526.438635] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7529.947953] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7537.583515] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7543.185858] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7551.145534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7551.527640] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7551.534419] LustreError: 193583:0:(llog.c:1696:llog_backup()) MGC192.168.203.139@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7551.548983] Lustre: 193583:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.203.139@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7551.730570] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7558.915832] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7569.085510] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7572.450944] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7572.457297] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7572.466055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7575.522047] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7575.522448] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7575.561161] Lustre: Skipped 1 previous similar message [ 7578.051842] Lustre: server umount lustre-MDT0000 complete [ 7580.649152] LustreError: 193608:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7580.673714] LustreError: 193608:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7581.514972] LustreError: 191701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851499 with bad export cookie 12114372452085067094 [ 7581.519200] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7581.527838] LustreError: 191701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7581.911710] Lustre: server umount lustre-MDT0001 complete [ 7595.845566] Lustre: server umount lustre-OST0000 complete [ 7604.428781] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 7615.166333] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 7661.581594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7662.097677] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7662.120837] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7662.174982] Lustre: lustre-MDT0000: new disk, initializing [ 7662.241746] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7662.259779] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7668.721356] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7681.861965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7682.028446] Lustre: 197114:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7682.078551] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7682.081793] Lustre: Skipped 1 previous similar message [ 7682.084879] LustreError: 197897:0:(llog.c:1696:llog_backup()) MGC192.168.203.139@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7682.092358] LustreError: 197897:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7682.097117] Lustre: 197897:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.139@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7682.131304] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7682.135169] Lustre: Skipped 1 previous similar message [ 7682.208275] Lustre: lustre-MDT0001: new disk, initializing [ 7682.303816] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7682.355953] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7682.367800] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7687.279405] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7698.089882] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7703.828409] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7710.683824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7710.986953] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7710.988953] Lustre: Skipped 1 previous similar message [ 7710.991705] LustreError: 199059:0:(llog.c:1696:llog_backup()) MGC192.168.203.139@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7711.002081] LustreError: 199059:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7711.006547] Lustre: 199059:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.139@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7711.020536] Lustre: 199059:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7711.060349] Lustre: lustre-OST0000: new disk, initializing [ 7711.063701] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7711.071279] Lustre: 199083:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7711.181092] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7717.099728] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7718.975041] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7718.987568] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7719.116748] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7727.413867] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7733.632297] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7734.052967] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7740.477890] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7740.790648] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7744.992735] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7744.997858] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7745.005975] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7745.020997] Lustre: Skipped 1 previous similar message [ 7745.038032] Lustre: Skipped 3 previous similar messages [ 7750.114800] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7750.118869] Lustre: Skipped 1 previous similar message [ 7755.240303] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7755.250364] Lustre: Skipped 1 previous similar message [ 7757.279263] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7757.524973] Lustre: server umount lustre-OST0000 complete [ 7764.451301] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7764.459347] LustreError: Skipped 1 previous similar message [ 7764.469018] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7764.484428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7767.481805] Lustre: server umount lustre-MDT0000 complete [ 7770.597205] LustreError: 197126:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7770.627977] LustreError: 197126:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7770.907336] LustreError: 197107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851688 with bad export cookie 12114372452085067815 [ 7770.922749] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7770.929040] LustreError: 197107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7771.229877] Lustre: server umount lustre-MDT0001 complete [ 7779.783967] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 03:14:55 (1788851695) [ 7796.148423] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 7804.422533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7804.785409] LustreError: 202213:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7804.850767] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7808.460179] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7810.020113] LustreError: 202214:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7815.372464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7819.699672] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7822.408914] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7828.948872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7835.333846] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7843.367725] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7843.520601] Lustre: lustre-OST0001: new disk, initializing [ 7843.524706] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7843.530205] Lustre: 204520:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7843.623094] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7843.632905] Lustre: Skipped 2 previous similar messages [ 7845.529820] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7845.536933] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7845.614781] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7849.861699] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7859.683266] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7864.236259] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7866.755612] Lustre: DEBUG MARKER: ADD OST3 [ 7872.127230] loop: module loaded [ 7872.201647] loop0: detected capacity change from 0 to 204800000 [ 7872.368298] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7872.386713] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7872.400341] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7878.701177] loop0: detected capacity change from 0 to 204800000 [ 7879.389966] Lustre: DEBUG MARKER: STOP OST3 [ 7884.983990] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7885.267511] Lustre: lustre-OST0003: new disk, initializing [ 7885.284904] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7885.297600] Lustre: 206544:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7887.008485] Lustre: *** cfs_fail_loc=2109, val=2*** [ 7887.178746] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7887.194145] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7887.233881] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 7889.065024] LustreError: 206689:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 7889.159766] Lustre: Failing over lustre-OST0003 [ 7889.429811] Lustre: server umount lustre-OST0003 complete [ 7891.950773] LustreError: 203701:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.203.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7891.986847] LustreError: 203701:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7892.467487] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7892.488751] Lustre: Skipped 1 previous similar message [ 7892.501106] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7893.252548] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7901.532937] Lustre: DEBUG MARKER: START OST3 again [ 7908.741836] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7908.993891] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 7909.003650] Lustre: Skipped 1 previous similar message [ 7909.015311] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 7910.343528] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 7910.722835] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7910.722848] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 7915.424253] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7935.979567] Lustre: Failing over lustre-OST0003 [ 7936.084248] Lustre: server umount lustre-OST0003 complete [ 7936.480247] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7936.485879] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7936.497306] LustreError: Skipped 1 previous similar message [ 7936.498259] LustreError: 203700:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7936.509832] Lustre: Skipped 2 previous similar messages [ 7936.543493] LustreError: 203700:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 7953.889321] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7953.895052] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7953.918968] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7953.926714] Lustre: Skipped 2 previous similar messages [ 7958.826936] Lustre: server umount lustre-MDT0000 complete [ 7962.045497] LustreError: 202195:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851879 with bad export cookie 12114372452085068697 [ 7962.052849] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7962.063361] LustreError: 202195:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7968.641363] Lustre: server umount lustre-MDT0001 complete [ 7972.633389] Lustre: server umount lustre-OST0000 complete [ 7976.824612] Lustre: server umount lustre-OST0001 complete [ 7983.779270] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 7990.538200] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 8034.830929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8035.326426] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8035.382120] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8035.620834] Lustre: lustre-MDT0000: new disk, initializing [ 8035.821733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8040.701744] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8052.004461] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8052.162514] Lustre: 211985:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8052.256779] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8052.270343] Lustre: Skipped 1 previous similar message [ 8052.367149] Lustre: lustre-MDT0001: new disk, initializing [ 8052.446170] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8052.450190] Lustre: Skipped 1 previous similar message [ 8052.485135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8052.500569] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8057.127840] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8067.730979] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8077.648143] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8088.028226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8088.617110] Lustre: lustre-OST0000: new disk, initializing [ 8088.637932] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8088.645754] Lustre: 213957:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8090.452731] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8090.476995] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8090.665881] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8095.227288] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8103.578703] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8108.018106] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8108.308345] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8112.959846] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8113.168259] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8116.203131] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8116.208090] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8116.212539] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8116.212547] Lustre: Skipped 8 previous similar messages [ 8116.216031] Lustre: Skipped 6 previous similar messages [ 8116.232808] LustreError: Skipped 1 previous similar message [ 8121.024597] Lustre: server umount lustre-OST0000 complete [ 8130.759417] Lustre: server umount lustre-MDT0000 complete [ 8131.565711] LustreError: 211998:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8131.578154] LustreError: 211998:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 17 previous similar messages [ 8134.140823] LustreError: 212780:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852051 with bad export cookie 12114372452085070993 [ 8134.147046] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8134.150904] LustreError: 212780:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8134.573291] Lustre: server umount lustre-MDT0001 complete [ 8138.475171] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 03:20:54 (1788852054) [ 8154.606957] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 8165.939480] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 8201.018048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8201.197789] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8201.233456] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8201.357347] Lustre: lustre-MDT0000: new disk, initializing [ 8201.432431] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8205.279744] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8215.289409] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8215.493574] Lustre: 218566:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8215.866599] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8219.623971] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8228.292586] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8234.029404] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8241.794895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8242.070990] Lustre: 220535:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8243.542396] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8247.394580] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8256.897947] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8263.182943] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8263.473398] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8269.544719] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8269.851408] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8274.401395] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8274.405048] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8274.418032] LustreError: Skipped 1 previous similar message [ 8274.418944] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8274.418949] Lustre: Skipped 3 previous similar messages [ 8274.454473] Lustre: Skipped 3 previous similar messages [ 8278.509195] Lustre: server umount lustre-OST0000 complete [ 8289.885517] Lustre: server umount lustre-MDT0000 complete [ 8294.037900] LustreError: 218557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852211 with bad export cookie 12114372452085071875 [ 8294.042778] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8294.049746] LustreError: 218557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8294.311233] Lustre: server umount lustre-MDT0001 complete [ 8302.660472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8303.071620] LustreError: 222750:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8307.036203] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8315.123408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8315.588519] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8315.596350] Lustre: Skipped 5 previous similar messages [ 8320.293876] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8329.927900] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8336.305148] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8344.038472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8350.863659] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8359.495247] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8371.173778] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8376.520697] Lustre: server umount lustre-OST0000 complete [ 8388.408136] Lustre: server umount lustre-MDT0000 complete [ 8392.335623] LustreError: 222731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852310 with bad export cookie 12114372452085072750 [ 8392.341309] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8392.350344] LustreError: 222731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8392.544832] Lustre: server umount lustre-MDT0001 complete [ 8497.503334] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 8500.509774] Key type lgssc unregistered [ 8500.864677] LNet: 226411:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8500.870757] LNetError: 226411:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8500.899325] LNet: Removed LNI 192.168.203.139@tcp [ 8501.795259] Key type .llcrypt unregistered [ 8501.798837] Key type ._llcrypt unregistered [ 8528.988605] Key type ._llcrypt registered [ 8528.995296] Key type .llcrypt registered [ 8529.130737] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 8547.113373] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 8548.250016] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8548.286425] alg: No test for adler32 (adler32-zlib) [ 8549.299726] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 8549.549591] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 8551.175422] Key type lgssc registered [ 8552.248581] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8590.457467] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8590.480909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8591.756711] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8591.821776] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8591.905437] Lustre: lustre-MDT0000: new disk, initializing [ 8592.020971] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8592.050171] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8596.427495] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8607.251537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8607.458523] Lustre: 229761:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8607.540835] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8607.548501] Lustre: Skipped 1 previous similar message [ 8607.644940] Lustre: lustre-MDT0001: new disk, initializing [ 8607.706780] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8607.750782] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8607.763451] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8613.071745] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8624.302471] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8633.422188] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8648.476332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8649.066242] Lustre: lustre-OST0000: new disk, initializing [ 8649.080126] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8649.108695] Lustre: 231731:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8649.254530] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8657.114155] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8657.139885] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8657.465524] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8659.195756] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8668.835832] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8673.576605] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8673.758181] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8680.723538] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8681.211530] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8688.128684] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8688.135095] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8688.166208] LustreError: Skipped 1 previous similar message [ 8688.172274] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8688.192154] Lustre: Skipped 1 previous similar message [ 8693.217794] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8693.244448] Lustre: Skipped 1 previous similar message [ 8698.335233] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8698.344819] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8698.356068] Lustre: Skipped 1 previous similar message [ 8698.477855] Lustre: server umount lustre-OST0000 complete [ 8703.462870] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8703.472947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8703.490798] Lustre: Skipped 2 previous similar messages [ 8708.690519] Lustre: server umount lustre-MDT0000 complete [ 8712.247795] LustreError: 229754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852630 with bad export cookie 4443061896180664302 [ 8712.251978] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8712.261938] LustreError: 229754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8712.596902] Lustre: server umount lustre-MDT0001 complete [ 8730.164302] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 8742.608866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8743.043876] LustreError: 234668:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8743.054330] LustreError: 234668:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8743.092095] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8748.518883] LustreError: 234669:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8748.840932] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8752.622217] LustreError: 234668:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8757.736157] LustreError: 234669:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8759.323476] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8759.847962] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8764.016049] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8767.977932] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8776.265987] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8776.890874] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8784.079976] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8793.930867] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8794.075159] Lustre: lustre-OST0001: new disk, initializing [ 8794.078888] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8794.087148] Lustre: 236978:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8794.174152] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8800.371475] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8800.379962] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8800.524291] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8801.383874] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8812.453769] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8819.655869] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8835.976410] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 03:32:30 (1788852750) [ 8841.700954] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8841.707049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8841.711250] Lustre: Skipped 2 previous similar messages [ 8841.717990] Lustre: Skipped 2 previous similar messages [ 8841.723382] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8844.784407] Lustre: server umount lustre-MDT0000 complete [ 8846.820312] LustreError: 234668:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8846.833101] LustreError: 234668:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8848.013152] LustreError: 234649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852765 with bad export cookie 4443061896180665184 [ 8848.019773] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8848.027214] LustreError: 234649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8848.295361] Lustre: server umount lustre-MDT0001 complete [ 8862.400622] Lustre: server umount lustre-OST0000 complete [ 8876.285558] Lustre: server umount lustre-OST0001 complete [ 8883.094813] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 8893.131425] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 8927.365257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8927.594211] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8927.631871] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8927.712997] Lustre: lustre-MDT0000: new disk, initializing [ 8927.798072] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8927.819412] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8932.831404] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8944.175713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8944.304946] Lustre: 240924:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8944.354321] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8944.358613] Lustre: Skipped 1 previous similar message [ 8944.449665] Lustre: lustre-MDT0001: new disk, initializing [ 8944.574560] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8944.587255] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8949.000903] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8956.784164] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8962.155645] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8968.409156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8968.877882] Lustre: lustre-OST0000: new disk, initializing [ 8968.904499] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8968.926610] Lustre: 242895:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8970.783168] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8970.805181] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8970.908456] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8975.359537] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8987.873874] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8996.112922] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8996.626915] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9002.979848] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9003.281242] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9007.076871] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9007.092557] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9007.111687] Lustre: Skipped 1 previous similar message [ 9007.128884] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9011.682829] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9011.716140] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9012.976695] Lustre: server umount lustre-OST0000 complete [ 9021.412494] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9021.424855] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9023.813184] Lustre: server umount lustre-MDT0000 complete [ 9027.040625] LustreError: 240933:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9027.065334] LustreError: 240933:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9027.253488] LustreError: 240917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852945 with bad export cookie 4443061896180667319 [ 9027.255867] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9027.261852] LustreError: 240917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9027.552321] Lustre: server umount lustre-MDT0001 complete [ 9035.567578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9036.227754] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9036.238432] Lustre: Skipped 2 previous similar messages [ 9040.665168] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9045.481217] LustreError: 245112:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9045.502996] LustreError: 245112:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9049.636494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9053.763997] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9061.632061] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9068.207511] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9077.652491] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9085.321955] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9093.490111] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9237.991191] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9237.991528] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9238.007299] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9238.007307] Lustre: Skipped 4 previous similar messages [ 9238.044177] Lustre: Skipped 2 previous similar messages [ 9252.319474] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9252.443225] Lustre: server umount lustre-OST0000 complete [ 9262.275641] Lustre: server umount lustre-MDT0000 complete [ 9263.585890] LustreError: 247625:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9263.605369] LustreError: 247625:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9265.614758] LustreError: 245092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788853183 with bad export cookie 4443061896180668201 [ 9265.618138] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9265.630217] LustreError: 245092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9265.960651] Lustre: server umount lustre-MDT0001 complete [ 9363.404083] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 9366.229893] Key type lgssc unregistered [ 9366.544883] LNet: 248721:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9366.555536] LNetError: 248721:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9366.578722] LNet: Removed LNI 192.168.203.139@tcp [ 9367.427620] Key type .llcrypt unregistered [ 9367.432963] Key type ._llcrypt unregistered [ 9395.721745] Key type ._llcrypt registered [ 9395.729506] Key type .llcrypt registered [ 9395.881461] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 9414.662082] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 9415.970186] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9416.124044] alg: No test for adler32 (adler32-zlib) [ 9417.423948] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 9417.830823] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 9419.559155] Key type lgssc registered [ 9420.756092] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9466.737243] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 9479.421102] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9479.463416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9480.845976] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9480.887886] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9480.994610] Lustre: lustre-MDT0000: new disk, initializing [ 9481.150735] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9481.167674] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9488.590207] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9501.276517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9501.348956] Lustre: 252788:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9501.419587] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9501.422687] Lustre: Skipped 1 previous similar message [ 9501.473280] Lustre: lustre-MDT0001: new disk, initializing [ 9501.529733] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9501.553728] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9501.559629] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9505.261675] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9510.208580] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9519.866518] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9520.265154] Lustre: lustre-OST0000: new disk, initializing [ 9520.286148] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9520.313756] Lustre: 254428:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9520.447267] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9520.809574] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9520.818705] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9520.977852] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9529.099058] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9543.509287] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9543.814427] Lustre: lustre-OST0001: new disk, initializing [ 9543.828448] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9543.845234] Lustre: 255298:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9544.040425] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9550.464715] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9550.471821] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9550.640591] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9551.600843] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9563.196578] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9573.295691] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9588.705919] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9588.733122] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9588.758545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9591.804759] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9591.806502] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9591.825223] Lustre: Skipped 1 previous similar message [ 9591.838294] Lustre: Skipped 3 previous similar messages [ 9593.952899] Lustre: server umount lustre-MDT0000 complete [ 9597.879468] LustreError: 252782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788853515 with bad export cookie 16203380043661647486 [ 9597.884432] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9597.896787] LustreError: 252782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9598.425632] Lustre: server umount lustre-MDT0001 complete [ 9612.897736] Lustre: server umount lustre-OST0000 complete [ 9626.758718] Lustre: server umount lustre-OST0001 complete [ 9640.737689] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9650.656967] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9660.246799] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9672.440859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9692.591788] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 9703.710059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9703.764396] Lustre: lustre-MDT0000: reset Object Index mappings [ 9704.452747] LustreError: 260313:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9704.474234] LustreError: 260313:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 9704.516491] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9709.539557] LustreError: 260314:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9710.491610] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9714.665441] LustreError: 260313:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9719.777606] LustreError: 260314:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9721.780618] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9721.796576] Lustre: lustre-MDT0001: reset Object Index mappings [ 9722.255521] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9727.137615] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9731.320900] Lustre: 261457:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9742.071506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9742.847373] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9746.981312] LustreError: 261812:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9750.103670] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9750.291701] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9755.109976] LustreError: 261814:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9755.149066] LustreError: 261814:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9759.418364] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9764.852098] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9766.028538] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9773.381524] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9776.998285] Lustre: 263331:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9783.777675] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9783.787699] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9783.793230] Lustre: Skipped 1 previous similar message [ 9783.800114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9784.033940] Lustre: server umount lustre-MDT0000 complete [ 9785.325339] LustreError: 260308:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9785.350446] LustreError: 260308:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9787.978871] LustreError: 260295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788853705 with bad export cookie 16203380043661654276 [ 9787.981222] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9788.005993] LustreError: 260295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9788.354778] Lustre: server umount lustre-MDT0001 complete [ 9803.117860] Lustre: server umount lustre-OST0000 complete [ 9807.073756] Lustre: 250297:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788853708/real 1788853708] req@ffff9deb48a59880 x1875748674737920/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788853724 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9807.100861] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9807.108402] Lustre: Skipped 3 previous similar messages [ 9807.995405] Lustre: server umount lustre-OST0001 complete [ 9832.022765] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 03:49:06 (1788853746) [ 9853.878253] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 9867.322704] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 9915.887457] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9916.224179] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9916.270769] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9916.403847] Lustre: lustre-MDT0000: new disk, initializing [ 9916.534244] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9916.544786] Lustre: Skipped 1 previous similar message [ 9916.581752] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9921.118135] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9931.422470] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9931.543308] Lustre: 267587:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9931.588699] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9931.592838] Lustre: Skipped 1 previous similar message [ 9931.664509] Lustre: lustre-MDT0001: new disk, initializing [ 9931.817393] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9931.825828] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9936.100922] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9945.904206] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9952.900909] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9959.307222] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9959.613080] Lustre: lustre-OST0000: new disk, initializing [ 9959.624023] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9959.634337] Lustre: 269557:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9961.174790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9961.186900] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9961.308619] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9966.227889] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9977.110891] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9982.973650] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9983.372621] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9989.084463] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9989.307625] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9992.168858] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9992.169932] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9992.176356] LustreError: Skipped 1 previous similar message [ 9992.196540] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9997.112612] Lustre: server umount lustre-OST0000 complete [10002.406990] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10002.415661] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10002.422294] Lustre: Skipped 2 previous similar messages [10002.442268] Lustre: Skipped 2 previous similar messages [10006.770447] Lustre: server umount lustre-MDT0000 complete [10007.524270] LustreError: 268457:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10007.556573] LustreError: 268457:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10010.418781] LustreError: 267577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788853928 with bad export cookie 16203380043661656376 [10010.427876] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10010.430598] LustreError: 267577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10010.690623] Lustre: server umount lustre-MDT0001 complete [10033.354480] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [10046.131643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10046.670818] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10046.683625] Lustre: Skipped 2 previous similar messages [10052.531900] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10060.670748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10065.858322] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10068.920687] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10078.077453] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10087.017394] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10097.966306] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10098.311761] Lustre: lustre-OST0001: new disk, initializing [10098.314807] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10098.318725] Lustre: 274802:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10099.721843] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10099.741841] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10099.908188] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10104.674301] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10114.860965] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10119.806647] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10126.439273] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10126.744195] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10138.080521] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10138.100514] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10138.137183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10143.348737] Lustre: server umount lustre-MDT0000 complete [10146.277222] LustreError: 273221:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10146.301527] LustreError: 273221:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [10147.832826] LustreError: 273228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788854065 with bad export cookie 16203380043661657258 [10147.834364] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10147.856721] LustreError: 273228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10148.217553] Lustre: server umount lustre-MDT0001 complete [10164.145376] Lustre: server umount lustre-OST0000 complete [10167.268871] Lustre: 250298:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788854069/real 1788854069] req@ffff9deb50448700 x1875748674851968/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788854085 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10168.802718] Lustre: 250299:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788854070/real 1788854070] req@ffff9dec768c0e00 x1875748674852224/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788854086 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10169.079791] Lustre: server umount lustre-OST0001 complete [10187.630695] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [10198.380528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10198.991679] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10198.998917] Lustre: Skipped 3 previous similar messages [10203.678804] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10213.412994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10217.904598] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10220.388285] Lustre: 278998:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10226.122146] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10232.981060] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10241.006139] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10243.473740] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10243.505720] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10248.402905] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10258.745490] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10264.075583] Lustre: 280875:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10269.923230] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10270.021506] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10280.168443] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 03:56:36 (1788854196) [10299.882046] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10299.886864] Lustre: Skipped 4 previous similar messages [10299.889730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10299.913526] Lustre: Skipped 7 previous similar messages [10302.740312] Lustre: server umount lustre-MDT0000 complete [10305.006181] LustreError: 281474:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10305.052991] LustreError: 281474:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [10307.166887] LustreError: 278601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788854224 with bad export cookie 16203380043661663796 [10307.171940] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10307.179039] LustreError: 278601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10313.819836] Lustre: server umount lustre-MDT0001 complete [10325.229918] Lustre: server umount lustre-OST0000 complete [10329.885351] Lustre: server umount lustre-OST0001 complete [10338.094156] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [10352.917305] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [10398.478570] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [10409.892806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10410.246406] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10410.299242] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10410.499942] Lustre: lustre-MDT0000: new disk, initializing [10410.796079] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10415.835311] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10427.985110] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10428.087680] Lustre: 285136:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10428.369792] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10433.847779] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10448.641183] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10449.161543] Lustre: 286771:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10450.586586] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10458.089092] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10469.396885] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10469.615817] Lustre: 287642:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10469.708307] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10469.720670] Lustre: Skipped 6 previous similar messages [10471.031296] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10471.044293] Lustre: Skipped 1 previous similar message [10471.262127] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10477.117193] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10487.625156] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10491.728782] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10491.732223] Lustre: Skipped 1 previous similar message [10512.965725] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 04:00:26 (1788854426) [10517.585458] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10517.599820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10517.609273] Lustre: Skipped 4 previous similar messages [10517.636389] Lustre: Skipped 7 previous similar messages [10523.012962] Lustre: server umount lustre-MDT0000 complete [10528.072059] LustreError: 285131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788854445 with bad export cookie 16203380043661671447 [10528.094483] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10528.506155] Lustre: server umount lustre-MDT0001 complete [10542.367806] Lustre: server umount lustre-OST0000 complete [10557.992204] Lustre: server umount lustre-OST0001 complete [10571.699247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10572.369363] LustreError: 289902:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10572.413453] LustreError: 289902:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [10579.334534] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10589.858542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10594.993256] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10606.222761] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10615.497734] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10623.940844] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10631.037342] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10640.471695] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10662.898542] loop1: detected capacity change from 0 to 204800000 [10663.158492] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10663.169896] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10663.194583] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10669.175849] loop1: detected capacity change from 0 to 204800000 [10675.424912] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10675.486878] LustreError: 289890:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs21075: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10675.507531] LustreError: 289890:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs21075 [10675.513492] LustreError: 289890:0:(mgs_handler.c:517:mgs_target_reg()) fs21075-MDT0001: Error checking target index: rc = -13 [10675.529548] LustreError: fs21075-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10675.545860] LustreError: 293625:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -13 [10675.553290] LustreError: 293625:0:(tgt_mount.c:2132:server_put_super()) no obd fs21075-MDT0001 [10675.561383] LustreError: 293625:0:(tgt_mount.c:116:server_deregister_mount()) fs21075-MDT0001 not registered [10675.605893] Lustre: server umount fs21075-MDT0001 complete [10675.616446] LustreError: 293625:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10684.424850] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10684.533516] Lustre: Setting parameter fs21075-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs21075-MDT0001 [10684.643391] Lustre: srv-fs21075-MDT0001: No data found on store. Initialize space. [10684.650077] Lustre: Skipped 4 previous similar messages [10684.772718] Lustre: fs21075-MDT0001: new disk, initializing [10684.782929] Lustre: Skipped 3 previous similar messages [10684.977099] Lustre: srv-fs21075-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10684.986094] Lustre: 294093:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs21075-MDT0001-mdtlov: can't create new llog: rc = -115 [10684.993392] LustreError: 294093:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs21075-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10689.691413] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10697.087317] Lustre: server umount fs21075-MDT0001 complete [10713.013903] loop1: detected capacity change from 0 to 204800000 [10714.603776] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10718.841749] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10719.366986] Lustre: 295659:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs21075-MDT0001-mdtlov: can't create new llog: rc = -115 [10719.372680] LustreError: 295659:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs21075-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10723.442921] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10728.614496] Lustre: server umount fs21075-MDT0001 complete [10743.776549] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10743.789095] LustreError: Skipped 1 previous similar message [10743.789775] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.39@tcp (stopping) [10743.794302] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10743.794311] Lustre: Skipped 2 previous similar messages [10743.862555] Lustre: Skipped 5 previous similar messages [10746.579322] Lustre: server umount lustre-MDT0000 complete [10756.001992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10756.150259] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10761.110636] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10761.730054] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10770.529832] loop1: detected capacity change from 0 to 204800000 [10776.529811] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10776.922337] Lustre: 298103:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs21075-MDT0001-mdtlov: can't create new llog: rc = -115 [10776.930865] LustreError: 298103:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs21075-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10781.385907] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10786.379893] Lustre: server umount fs21075-MDT0001 complete [10812.905585] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10823.136807] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10823.411303] Lustre: server umount lustre-OST0000 complete [10834.512143] Lustre: server umount lustre-MDT0000 complete [10838.421389] LustreError: 289883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788854756 with bad export cookie 16203380043661674128 [10838.430966] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10838.436980] LustreError: 289883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [10839.008838] Lustre: server umount lustre-MDT0001 complete [10850.553707] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [10853.444683] Key type lgssc unregistered [10853.762621] LNet: 300449:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10853.776820] LNetError: 300449:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10853.807255] LNet: Removed LNI 192.168.203.139@tcp [10854.721128] Key type .llcrypt unregistered [10854.723279] Key type ._llcrypt unregistered [10876.767883] Key type ._llcrypt registered [10876.769860] Key type .llcrypt registered [10877.003257] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [10878.510338] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10878.548379] alg: No test for adler32 (adler32-zlib) [10880.247374] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [10880.640813] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [10882.383151] Key type lgssc registered [10884.536450] Lustre: Echo OBD driver; http://www.lustre.org/ [10897.361775] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10897.400220] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10899.047209] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10904.330230] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10909.157046] LustreError: 301717:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10909.186353] LustreError: 301717:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10912.800780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10913.153647] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10917.308626] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10924.455868] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10930.578698] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10938.263932] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10938.943762] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10944.050378] LustreError: 303546:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10947.544418] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10949.109131] LustreError: 303546:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10954.259443] LustreError: 303772:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10959.128951] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10960.743747] LustreError: 303772:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10960.783381] LustreError: 303772:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10971.078218] LustreError: 303547:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10971.118842] LustreError: 303547:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10979.128764] loop1: detected capacity change from 0 to 204800000 [10979.445860] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10979.492255] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10979.503181] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10985.948499] loop1: detected capacity change from 0 to 204800000 [10988.009126] LustreError: 303772:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10988.045534] LustreError: 303772:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [10992.466332] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10992.650250] Lustre: fs7499-OST0000: new disk, initializing [10992.653879] Lustre: srv-fs7499-OST0000: No data found on store. Initialize space. [10992.659303] Lustre: 305454:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10992.718168] Lustre: fs7499-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11000.910436] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [11010.537517] Lustre: server umount fs7499-OST0000 complete [11022.255531] LustreError: 303547:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11022.276635] LustreError: 303547:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 18 previous similar messages [11022.651882] loop1: detected capacity change from 0 to 204800000 [11023.847420] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11027.487256] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11027.635812] Lustre: fs7499-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11033.465629] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [11039.209357] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11042.503386] loop2: detected capacity change from 0 to 204800000 [11042.786137] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11042.804401] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11042.815729] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11048.648334] loop2: detected capacity change from 0 to 204800000 [11054.850327] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11054.961895] LustreError: fs7499-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11054.997702] LustreError: fs7499-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11055.021896] LustreError: 308496:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -13 [11055.038722] LustreError: 308496:0:(tgt_mount.c:2132:server_put_super()) no obd fs7499-OST0001 [11055.049888] LustreError: 308496:0:(tgt_mount.c:116:server_deregister_mount()) fs7499-OST0001 not registered [11055.108275] Lustre: server umount fs7499-OST0001 complete [11055.114493] LustreError: 308496:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11065.297096] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11065.445213] Lustre: fs7499-OST0001: new disk, initializing [11065.450452] Lustre: srv-fs7499-OST0001: No data found on store. Initialize space. [11065.457757] Lustre: 309048:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11065.554455] Lustre: fs7499-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11073.619480] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [11082.716771] Lustre: server umount fs7499-OST0000 complete [11088.817116] LustreError: 303546:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11088.849241] LustreError: 303546:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 38 previous similar messages [11093.426115] Lustre: server umount fs7499-OST0001 complete [11106.725323] loop1: detected capacity change from 0 to 204800000 [11112.842276] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11113.246302] Lustre: fs7499-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11119.750391] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [11126.828646] loop2: detected capacity change from 0 to 204800000 [11133.144297] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11140.322635] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [11150.972271] Lustre: server umount fs7499-OST0000 complete [11164.344383] Lustre: server umount fs7499-OST0001 complete [11175.407227] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11179.495319] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11179.506057] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11179.517256] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11180.525420] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11180.536201] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11185.668346] Lustre: server umount lustre-OST0000 complete [11190.752667] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11190.766492] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11190.782774] Lustre: Skipped 1 previous similar message [11190.786069] Lustre: Skipped 1 previous similar message [11195.880671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11195.894525] Lustre: Skipped 2 previous similar messages [11196.928722] Lustre: server umount lustre-MDT0000 complete [11201.438947] LustreError: 301697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788855119 with bad export cookie 14748148283060488658 [11201.448789] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11201.705738] Lustre: server umount lustre-MDT0001 complete [11219.085959] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [11223.185233] Key type lgssc unregistered [11223.602883] LNet: 314990:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11223.615082] LNetError: 314990:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11223.655102] LNet: Removed LNI 192.168.203.139@tcp [11224.808121] Key type .llcrypt unregistered [11224.821950] Key type ._llcrypt unregistered [11258.536109] Key type ._llcrypt registered [11258.539732] Key type .llcrypt registered [11258.708970] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [11274.973278] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [11276.171617] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11276.240423] alg: No test for adler32 (adler32-zlib) [11277.310773] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [11277.544498] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [11279.247272] Key type lgssc registered [11280.036315] Lustre: Echo OBD driver; http://www.lustre.org/ [11314.782719] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11314.828430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11316.219171] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11316.267161] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11316.391695] Lustre: lustre-MDT0000: new disk, initializing [11316.487207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11316.504561] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11322.057551] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [11335.064165] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11335.294636] Lustre: 318850:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11335.356811] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11335.362545] Lustre: Skipped 1 previous similar message [11335.458628] Lustre: lustre-MDT0001: new disk, initializing [11335.588937] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11335.644808] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11335.660830] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11343.203748] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [11356.316302] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11364.031397] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11373.427542] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11373.616402] Lustre: lustre-OST0000: new disk, initializing [11373.619401] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11373.623547] Lustre: 320818:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11373.705896] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11381.712566] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [11383.412789] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11383.444282] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11383.535387] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11391.953622] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11393.998913] Lustre: Mounted lustre-client - version 2.17.57_111_g33f3974 [11394.979184] Lustre: Unmounted lustre-client [11409.380599] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11409.382245] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11409.401611] LustreError: Skipped 1 previous similar message [11409.402616] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11409.445709] Lustre: Skipped 1 previous similar message [11412.275031] Lustre: server umount lustre-OST0000 complete [11417.575750] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11417.587177] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11417.599922] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11417.604564] Lustre: Skipped 1 previous similar message [11419.617144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11419.618914] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11422.543186] Lustre: server umount lustre-MDT0000 complete [11426.688373] LustreError: 318842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788855344 with bad export cookie 5790705075860755862 [11426.701586] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11427.087824] Lustre: server umount lustre-MDT0001 complete [11439.525052] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [11442.645106] Key type lgssc unregistered [11446.627428] LNet: 1 local NIs in recovery (showing 1): 192.168.203.139@tcp [11447.136168] LNet: 322977:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11447.148900] LNetError: 322977:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11523.942434] LNet: Removed LNI 192.168.203.139@tcp [11525.467193] Key type .llcrypt unregistered [11525.471277] Key type ._llcrypt unregistered [11547.060344] Key type ._llcrypt registered [11547.062860] Key type .llcrypt registered [11547.233199] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [11565.963199] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [11567.127900] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11567.321632] alg: No test for adler32 (adler32-zlib) [11568.404899] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [11568.780216] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [11570.527615] Key type lgssc registered [11571.728827] Lustre: Echo OBD driver; http://www.lustre.org/ [11603.510874] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 04:18:38 (1788855518) [11610.707223] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11610.748316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11612.042474] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11612.082157] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11612.150702] Lustre: lustre-MDT0000: new disk, initializing [11612.204761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11612.220964] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11616.339990] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [11627.130937] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11627.304191] Lustre: 326200:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11627.362888] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11627.367536] Lustre: Skipped 1 previous similar message [11627.451352] Lustre: lustre-MDT0001: new disk, initializing [11627.561211] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11627.613856] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11627.634421] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11632.557034] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [11643.937687] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11650.444669] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11657.788464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11658.116255] Lustre: lustre-OST0000: new disk, initializing [11658.126534] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11658.141172] Lustre: 328166:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11658.255232] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11664.882638] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [11664.986847] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11664.999512] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11665.155095] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11676.051816] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11696.096469] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11696.104214] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11696.111373] LustreError: Skipped 1 previous similar message [11696.126092] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11696.141262] Lustre: Skipped 1 previous similar message [11698.828648] Lustre: server umount lustre-OST0000 complete [11704.292735] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11704.301298] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11704.310693] Lustre: Skipped 1 previous similar message [11704.313971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11706.339671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11706.342534] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11708.627239] LustreError: 329321:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (00000000a33723bc) refcount nonzero (2) after lock cleanup; forcing cleanup. [11710.743186] Lustre: server umount lustre-MDT0000 complete [11714.363895] LustreError: 326192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788855632 with bad export cookie 605829466766210498 [11714.386057] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11715.019053] Lustre: server umount lustre-MDT0001 complete [11729.195801] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [11732.890455] Key type lgssc unregistered [11733.354335] LNet: 330074:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11733.365320] LNetError: 330074:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11734.381070] LNet: Removed LNI 192.168.203.139@tcp [11735.682223] Key type .llcrypt unregistered [11735.683938] Key type ._llcrypt unregistered [11759.749448] Key type ._llcrypt registered [11759.751325] Key type .llcrypt registered [11759.914324] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [11761.558278] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11761.715077] alg: No test for adler32 (adler32-zlib) [11763.220512] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [11763.614216] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [11765.223181] Key type lgssc registered [11766.575180] Lustre: Echo OBD driver; http://www.lustre.org/ [11779.739226] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11779.776198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11781.568260] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11787.035084] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [11791.855122] LustreError: 331346:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11791.886119] LustreError: 331346:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11796.962514] LustreError: 331345:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11797.651372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11798.225473] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11803.179176] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [11811.073294] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11816.889359] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11823.760979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11824.008914] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11831.086095] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [11839.160313] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11853.800944] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11855.133559] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11859.936025] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11859.950054] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11859.967824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11862.477456] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.39@tcp (stopping) [11863.014086] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11863.052840] Lustre: Skipped 1 previous similar message [11867.566210] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.39@tcp (stopping) [11867.580858] Lustre: Skipped 3 previous similar messages [11872.693435] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.39@tcp (stopping) [11872.716836] Lustre: Skipped 3 previous similar messages [11874.271965] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11874.568084] Lustre: server umount lustre-MDT0000 complete [11877.811961] LustreError: 331341:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11878.891277] LustreError: 331326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788855796 with bad export cookie 2921393276421917445 [11878.892701] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11878.901834] LustreError: 331326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11882.927345] LustreError: 333786:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11882.950044] LustreError: 333786:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11882.970792] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.39@tcp (stopping) [11882.984187] Lustre: Skipped 3 previous similar messages [11883.496878] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11888.053772] LustreError: 331342:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11888.083624] LustreError: 331342:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11893.176358] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.39@tcp (stopping) [11893.186039] Lustre: Skipped 4 previous similar messages [11893.215149] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11893.527397] Lustre: server umount lustre-MDT0001 complete [11897.346359] Lustre: server umount lustre-OST0000 complete [11906.080901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11910.841390] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [11914.086615] Lustre: 335532:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.139@tcp [11916.393796] Lustre: server umount MGS complete [11925.014396] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11925.526884] LustreError: 336103:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11925.553967] LustreError: 336103:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11925.611993] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11929.891785] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [11938.369211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11938.607594] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11942.980905] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [11952.921549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11953.243101] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11962.327832] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [11970.617388] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11971.934484] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [11978.243571] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11980.086143] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11984.358911] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11984.370045] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11984.380815] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11985.895250] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [11990.142771] Lustre: server umount lustre-OST0000 complete [11995.108274] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11995.112619] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11995.126991] Lustre: Skipped 1 previous similar message [11999.896685] Lustre: server umount lustre-MDT0000 complete [12001.762378] LustreError: 336103:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12001.793057] LustreError: 336103:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12003.744758] LustreError: 336085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788855921 with bad export cookie 2921393276421918285 [12003.748130] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12003.753067] LustreError: 336085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12004.076987] Lustre: server umount lustre-MDT0001 complete [12014.782421] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [12017.601781] Key type lgssc unregistered [12017.854746] LNet: 339539:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12017.865576] LNetError: 339539:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12017.888605] LNet: Removed LNI 192.168.203.139@tcp [12018.823175] Key type .llcrypt unregistered [12018.826411] Key type ._llcrypt unregistered [12040.669462] Key type ._llcrypt registered [12040.671908] Key type .llcrypt registered [12040.807591] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [12057.764845] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [12059.005801] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12059.115822] alg: No test for adler32 (adler32-zlib) [12060.285894] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [12060.547277] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [12062.223279] Key type lgssc registered [12063.581727] Lustre: Echo OBD driver; http://www.lustre.org/ [12101.943836] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12101.978836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12103.376450] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12103.477490] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12103.674436] Lustre: lustre-MDT0000: new disk, initializing [12103.759410] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12103.778815] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12108.741963] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [12119.540329] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12119.636117] Lustre: 342698:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12119.686667] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12119.689309] Lustre: Skipped 1 previous similar message [12119.735962] Lustre: lustre-MDT0001: new disk, initializing [12119.809985] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12119.861503] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12119.876240] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12125.632359] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [12140.574847] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12150.325317] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12160.019525] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12160.616045] Lustre: lustre-OST0000: new disk, initializing [12160.626824] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12160.635167] Lustre: 344672:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12160.813020] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12168.554284] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [12169.336956] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12169.361210] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12169.593542] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12179.302554] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12184.863294] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12185.093244] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12190.285248] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12190.519498] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12195.298771] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12195.300176] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12195.302702] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12195.306563] LustreError: Skipped 1 previous similar message [12195.350342] Lustre: Skipped 1 previous similar message [12198.642854] Lustre: server umount lustre-OST0000 complete [12205.545147] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12205.550447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12205.561432] Lustre: Skipped 1 previous similar message [12205.564038] Lustre: Skipped 1 previous similar message [12209.108659] Lustre: server umount lustre-MDT0000 complete [12213.270219] LustreError: 342693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788856131 with bad export cookie 3701823451827089441 [12213.274420] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12213.285159] LustreError: 342693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12213.849745] Lustre: server umount lustre-MDT0001 complete [12223.481286] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 04:28:59 (1788856139) [12242.263950] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [12252.487027] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [12294.038622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12294.505931] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12294.564405] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12294.699884] Lustre: lustre-MDT0000: new disk, initializing [12294.901901] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12294.944412] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12299.979647] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [12312.712798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12313.014166] Lustre: 349433:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12313.166425] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12313.177239] Lustre: Skipped 1 previous similar message [12313.307402] Lustre: lustre-MDT0001: new disk, initializing [12313.506948] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12313.585251] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12313.602899] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12321.653369] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [12335.058213] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12340.984953] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12348.314503] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12348.666521] Lustre: lustre-OST0000: new disk, initializing [12348.676497] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12348.682863] Lustre: 351405:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12348.781246] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12354.623335] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12354.635667] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12354.784499] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12355.292877] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [12365.804374] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12401.139892] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12401.151815] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12401.156370] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12401.166226] LustreError: Skipped 1 previous similar message [12401.177465] Lustre: Skipped 2 previous similar messages [12405.179347] Lustre: server umount lustre-OST0000 complete [12410.853911] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12410.864742] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12410.887976] Lustre: Skipped 1 previous similar message [12410.894843] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12415.251797] Lustre: server umount lustre-MDT0000 complete [12416.486903] LustreError: 349443:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12416.505909] LustreError: 349443:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12419.321776] LustreError: 349425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788856337 with bad export cookie 3701823451827090323 [12419.321775] LustreError: 349424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788856337 with bad export cookie 3701823451827090323 [12419.322466] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12419.335724] LustreError: 349424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12419.670926] Lustre: server umount lustre-MDT0001 complete [12431.108414] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [12434.159271] Key type lgssc unregistered [12434.510994] LNet: 353579:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12434.520131] LNetError: 353579:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12434.536835] LNet: Removed LNI 192.168.203.139@tcp [12435.348813] Key type .llcrypt unregistered [12435.350842] Key type ._llcrypt unregistered [12553.002470] Key type ._llcrypt registered [12553.008612] Key type .llcrypt registered [12553.309107] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [12554.002480] Key type .llcrypt unregistered [12554.015537] Key type ._llcrypt unregistered [12642.723876] Key type ._llcrypt registered [12642.727853] Key type .llcrypt registered [12642.835370] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [12643.114438] Key type .llcrypt unregistered [12643.118798] Key type ._llcrypt unregistered [12675.064049] Key type ._llcrypt registered [12675.067642] Key type .llcrypt registered [12675.175314] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [12675.487416] Key type .llcrypt unregistered [12675.493468] Key type ._llcrypt unregistered [12703.455283] Key type ._llcrypt registered [12703.457338] Key type .llcrypt registered [12703.546937] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [12703.762376] Key type .llcrypt unregistered [12703.766994] Key type ._llcrypt unregistered [12778.492136] Key type ._llcrypt registered [12778.497297] Key type .llcrypt registered [12778.740677] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [12779.449537] Key type .llcrypt unregistered [12779.456851] Key type ._llcrypt unregistered [12816.617693] Key type ._llcrypt registered [12816.621107] Key type .llcrypt registered [12816.797773] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [12817.059394] Key type .llcrypt unregistered [12817.061893] Key type ._llcrypt unregistered [12846.492922] Key type ._llcrypt registered [12846.495705] Key type .llcrypt registered [12846.655202] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [12847.006478] Key type .llcrypt unregistered [12847.012206] Key type ._llcrypt unregistered [12881.362948] Key type ._llcrypt registered [12881.370121] Key type .llcrypt registered [12881.449591] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [12881.690205] Key type .llcrypt unregistered [12881.695321] Key type ._llcrypt unregistered [12915.834133] Key type ._llcrypt registered [12915.837716] Key type .llcrypt registered [12915.982553] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [12916.266807] Key type .llcrypt unregistered [12916.273868] Key type ._llcrypt unregistered [12956.679060] Key type ._llcrypt registered [12956.681352] Key type .llcrypt registered [12956.852787] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [12957.261729] Key type .llcrypt unregistered [12957.266277] Key type ._llcrypt unregistered [13032.374470] Key type ._llcrypt registered [13032.382233] Key type .llcrypt registered [13032.552234] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [13032.891554] Key type .llcrypt unregistered [13032.895717] Key type ._llcrypt unregistered [13068.896296] Key type ._llcrypt registered [13068.899566] Key type .llcrypt registered [13069.012398] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [13069.485434] Key type .llcrypt unregistered [13069.491956] Key type ._llcrypt unregistered [13104.700177] Key type ._llcrypt registered [13104.702539] Key type .llcrypt registered [13104.771644] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [13105.002775] Key type .llcrypt unregistered [13105.005815] Key type ._llcrypt unregistered [13169.451687] Key type ._llcrypt registered [13169.463489] Key type .llcrypt registered [13169.623647] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [13185.643235] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [13186.747384] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13186.764851] alg: No test for adler32 (adler32-zlib) [13188.174543] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [13188.521375] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [13190.215157] Key type lgssc registered [13191.552314] Lustre: Echo OBD driver; http://www.lustre.org/ [13235.653439] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13235.689259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13237.134741] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13237.157550] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13237.225670] Lustre: tst250fs-MDT0000: new disk, initializing [13237.371602] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13237.388621] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13243.670526] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [13257.659308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13257.858596] Lustre: 373878:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13257.916222] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13257.923027] Lustre: Skipped 1 previous similar message [13258.042594] Lustre: tst250fs-MDT0001: new disk, initializing [13258.222591] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13258.295892] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13258.313711] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13264.260215] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [13270.042994] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13279.864607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13280.436905] Lustre: tst250fs-OST0000: new disk, initializing [13280.457454] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13280.471953] Lustre: 375515:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13280.626735] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13287.020716] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13287.038950] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13287.248745] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13288.898233] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [13305.816575] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13306.174583] Lustre: tst250fs-OST0001: new disk, initializing [13306.181781] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13306.197826] Lustre: 376388:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13306.452624] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13314.715009] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13314.723525] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [13314.741728] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13314.869589] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13320.483254] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13323.378162] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [13323.384986] Lustre: Skipped 2 previous similar messages [13345.252528] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13345.267463] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13345.285459] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13347.075108] Lustre: server umount tst250fs-MDT0000 complete [13350.672067] LustreError: 376387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788857268 with bad export cookie 11519541448472362615 [13350.675730] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13350.687711] LustreError: 376387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13350.882789] Lustre: tst250fs-MDT0001-lwp-OST0000: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13350.898407] Lustre: Skipped 3 previous similar messages [13350.988027] Lustre: server umount tst250fs-MDT0001 complete [13357.700531] Lustre: server umount tst250fs-OST0000 complete [13362.042177] Lustre: server umount tst250fs-OST0001 complete [13382.692466] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [13392.085852] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [13428.829924] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 04:49:04 (1788857344) [13430.662190] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13448.780876] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [13461.182807] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [13497.785672] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13200 sec ========= 04:50:13 (1788857413) [13499.592682] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 04:50:15 (1788857415) === [13503.711369] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 04:50:19 (1788857419) === [13549.850417] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [13552.782671] Key type lgssc unregistered [13553.076662] LNet: 385537:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13553.080490] LNetError: 385537:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13553.102720] LNet: Removed LNI 192.168.203.139@tcp [13553.937083] Key type .llcrypt unregistered [13553.940934] Key type ._llcrypt unregistered