[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 492573353 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003267] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.008367] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010012] pid_max: default: 32768 minimum: 301 [ 0.011139] LSM: Security Framework initializing [ 0.012049] Yama: becoming mindful. [ 0.013034] SELinux: Initializing. [ 0.014065] *** VALIDATE selinux *** [ 0.023321] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027568] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029141] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030098] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032019] *** VALIDATE tmpfs *** [ 0.033432] *** VALIDATE proc *** [ 0.034211] *** VALIDATE cgroup *** [ 0.035020] *** VALIDATE cgroup2 *** [ 0.037173] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038139] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040028] Spectre V2 : User space: Vulnerable [ 0.041007] Speculative Store Bypass: Vulnerable [ 0.044220] debug: unmapping init [mem 0xffffffffb4259000-0xffffffffb4260fff] [ 0.046186] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047635] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048021] ... version: 2 [ 0.049011] ... bit width: 48 [ 0.050011] ... generic registers: 4 [ 0.051015] ... value mask: 0000ffffffffffff [ 0.052017] ... max period: 00007fffffffffff [ 0.053012] ... fixed-purpose events: 3 [ 0.054010] ... event mask: 000000070000000f [ 0.055304] rcu: Hierarchical SRCU implementation. [ 0.057309] smp: Bringing up secondary CPUs ... [ 0.058560] x86: Booting SMP configuration: [ 0.059028] .... node #0, CPUs: #1 #2 #3 [ 0.063021] smp: Brought up 1 node, 4 CPUs [ 0.065017] smpboot: Max logical packages: 1 [ 0.066011] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.256764] node 0 deferred pages initialised in 188ms [ 0.259187] devtmpfs: initialized [ 0.260217] x86/mm: Memory block size: 128MB [ 0.262734] gcov: version magic: 0x41383552 [ 0.264183] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.265109] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.266307] pinctrl core: initialized pinctrl subsystem [ 0.267235] [ 0.267898] ************************************************************* [ 0.268020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.269023] ** ** [ 0.270020] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.271022] ** ** [ 0.272016] ** This means that this kernel is built to expose internal ** [ 0.273023] ** IOMMU data structures, which may compromise security on ** [ 0.274017] ** your system. ** [ 0.275021] ** ** [ 0.276034] ** If you see this message and you are not debugging the ** [ 0.277015] ** kernel, report this immediately to your vendor! ** [ 0.278019] ** ** [ 0.279015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.280013] ************************************************************* [ 0.281762] NET: Registered protocol family 16 [ 0.282564] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.283074] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.284072] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.285539] cpuidle: using governor menu [ 0.287908] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.290604] PCI: Using configuration type 1 for base access [ 0.293141] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.303174] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.306039] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.310036] cryptd: max_cpu_qlen set to 1000 [ 0.313150] ACPI: Added _OSI(Module Device) [ 0.314023] ACPI: Added _OSI(Processor Device) [ 0.315020] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.317019] ACPI: Added _OSI(Processor Aggregator Device) [ 0.322442] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.329422] ACPI: Interpreter enabled [ 0.331075] ACPI: PM: (supports S0 S3 S4 S5) [ 0.332015] ACPI: Using IOAPIC for interrupt routing [ 0.334170] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.337891] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.347950] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.351059] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.353023] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.357113] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.362771] acpiphp: Slot [2] registered [ 0.365223] acpiphp: Slot [5] registered [ 0.367190] acpiphp: Slot [6] registered [ 0.369218] acpiphp: Slot [7] registered [ 0.371214] acpiphp: Slot [8] registered [ 0.373130] acpiphp: Slot [9] registered [ 0.374122] acpiphp: Slot [10] registered [ 0.376120] acpiphp: Slot [3] registered [ 0.378163] acpiphp: Slot [4] registered [ 0.379146] acpiphp: Slot [11] registered [ 0.381107] acpiphp: Slot [12] registered [ 0.383110] acpiphp: Slot [13] registered [ 0.384109] acpiphp: Slot [14] registered [ 0.386110] acpiphp: Slot [15] registered [ 0.388113] acpiphp: Slot [16] registered [ 0.389135] acpiphp: Slot [17] registered [ 0.391120] acpiphp: Slot [18] registered [ 0.392097] acpiphp: Slot [19] registered [ 0.394115] acpiphp: Slot [20] registered [ 0.395110] acpiphp: Slot [21] registered [ 0.397118] acpiphp: Slot [22] registered [ 0.399127] acpiphp: Slot [23] registered [ 0.400094] acpiphp: Slot [24] registered [ 0.401095] acpiphp: Slot [25] registered [ 0.403129] acpiphp: Slot [26] registered [ 0.405087] acpiphp: Slot [27] registered [ 0.406095] acpiphp: Slot [28] registered [ 0.408075] acpiphp: Slot [29] registered [ 0.409119] acpiphp: Slot [30] registered [ 0.411137] acpiphp: Slot [31] registered [ 0.412096] PCI host bridge to bus 0000:00 [ 0.414029] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.417034] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.419030] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.422030] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.425033] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.428039] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.430208] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.434212] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.437567] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.448770] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.453512] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.457026] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.459023] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.461018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.464632] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.466890] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.469039] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.472936] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.478015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.492016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.497009] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.502113] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.510020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.518022] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.535018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.547483] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.560015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.568016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.588014] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.598739] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.605013] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.611013] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.631014] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.640742] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.647012] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.653015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.672015] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.682593] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.689013] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.696014] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.713015] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.721415] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.727016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.734016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.750017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.762809] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.765425] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.768333] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.771392] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.773244] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.779032] iommu: Default domain type: Passthrough [ 0.780390] SCSI subsystem initialized [ 0.782141] ACPI: bus type USB registered [ 0.783104] usbcore: registered new interface driver usbfs [ 0.785086] usbcore: registered new interface driver hub [ 0.786075] usbcore: registered new device driver usb [ 0.788155] pps_core: LinuxPPS API ver. 1 registered [ 0.789011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.793060] PTP clock support registered [ 0.796086] EDAC MC: Ver: 3.0.0 [ 0.798149] PCI: Using ACPI for IRQ routing [ 0.800426] NetLabel: Initializing [ 0.802016] NetLabel: domain hash size = 128 [ 0.804016] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.806106] NetLabel: unlabeled traffic allowed by default [ 0.810203] vgaarb: loaded [ 0.812385] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.814023] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.822402] clocksource: Switched to clocksource kvm-clock [ 0.933201] VFS: Disk quotas dquot_6.6.0 [ 0.934962] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.937853] *** VALIDATE ramfs *** [ 0.939295] *** VALIDATE hugetlbfs *** [ 0.941857] pnp: PnP ACPI init [ 0.944590] pnp: PnP ACPI: found 6 devices [ 0.964818] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.968753] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.972017] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.974433] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.977104] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.979877] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.983056] NET: Registered protocol family 2 [ 0.985411] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.990451] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.994301] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.999644] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.004144] TCP: Hash tables configured (established 65536 bind 65536) [ 1.007538] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.011377] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.014044] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.016553] NET: Registered protocol family 1 [ 1.019272] RPC: Registered named UNIX socket transport module. [ 1.022345] RPC: Registered udp transport module. [ 1.023772] RPC: Registered tcp transport module. [ 1.025174] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.027264] NET: Registered protocol family 44 [ 1.029068] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.031140] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.033048] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.035077] PCI: CLS 0 bytes, default 64 [ 1.037133] Unpacking initramfs... [ 2.534374] debug: unmapping init [mem 0xffff953d7cc54000-0xffff953d7ffbffff] [ 2.538564] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.540956] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.544123] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.040288] Initialise system trusted keyrings [ 3.041857] Key type blacklist registered [ 3.044077] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.053719] zbud: loaded [ 3.057332] *** VALIDATE nfs *** [ 3.058869] *** VALIDATE nfs4 *** [ 3.060523] pstore: using deflate compression [ 3.064273] Platform Keyring initialized [ 3.180811] NET: Registered protocol family 38 [ 3.183221] Key type asymmetric registered [ 3.185126] Asymmetric key parser 'x509' registered [ 3.186962] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.189669] io scheduler mq-deadline registered [ 3.191443] io scheduler kyber registered [ 3.193341] io scheduler bfq registered [ 3.195597] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.199720] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.203717] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.206932] ACPI: Power Button [PWRF] [ 3.213060] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.220523] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.234944] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.242289] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.261346] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.288552] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.319205] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.326360] Non-volatile memory driver v1.3 [ 3.327821] Linux agpgart interface v0.103 [ 3.361151] virtio_blk virtio1: [vda] 145960 512-byte logical blocks (74.7 MB/71.3 MiB) [ 3.365028] vda: detected capacity change from 0 to 74731520 [ 3.387738] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.391117] vdb: detected capacity change from 0 to 1073741824 [ 3.413377] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.417638] vdc: detected capacity change from 0 to 2621440000 [ 3.433843] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.437293] vdd: detected capacity change from 0 to 2621440000 [ 3.453756] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.456948] vde: detected capacity change from 0 to 4294967296 [ 3.473566] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.476770] vdf: detected capacity change from 0 to 4294967296 [ 3.484198] libphy: Fixed MDIO Bus: probed [ 3.490241] usbcore: registered new interface driver usbserial_generic [ 3.492752] usbserial: USB Serial support registered for generic [ 3.494992] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.499499] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.501498] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.504052] mousedev: PS/2 mouse device common for all mice [ 3.507764] rtc_cmos 00:05: RTC can wake from S4 [ 3.511220] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.515602] rtc_cmos 00:05: registered as rtc0 [ 3.518735] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.518934] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.526273] intel_pstate: CPU model not supported [ 3.528678] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.532965] hid: raw HID events driver (C) Jiri Kosina [ 3.535208] usbcore: registered new interface driver usbhid [ 3.537256] usbhid: USB HID core driver [ 3.539198] drop_monitor: Initializing network drop monitor service [ 3.541593] Initializing XFRM netlink socket [ 3.543735] NET: Registered protocol family 10 [ 3.546756] Segment Routing with IPv6 [ 3.548265] NET: Registered protocol family 17 [ 3.550426] mpls_gso: MPLS GSO support [ 3.556744] RAS: Correctable Errors collector initialized. [ 3.559057] AVX version of gcm_enc/dec engaged. [ 3.560703] AES CTR mode by8 optimization enabled [ 3.633713] sched_clock: Marking stable (3633676449, 0)->(4595147234, -961470785) [ 3.637497] registered taskstats version 1 [ 3.639457] Loading compiled-in X.509 certificates [ 3.641493] zswap: loaded using pool lzo/zbud [ 3.664930] Key type big_key registered [ 3.678088] Key type encrypted registered [ 3.679780] ima: No TPM chip found, activating TPM-bypass! [ 3.681963] ima: Allocated hash algorithm: sha1 [ 3.683599] ima: No architecture policies found [ 3.685374] evm: Initialising EVM extended attributes: [ 3.687111] evm: security.selinux [ 3.688492] evm: security.ima [ 3.689520] evm: security.capability [ 3.690806] evm: HMAC attrs: 0x1 [ 3.693132] rtc_cmos 00:05: setting system clock to 2026-09-08 21:35:30 UTC (1788903330) [ 3.700806] debug: unmapping init [mem 0xffffffffb5203000-0xffffffffb53fffff] [ 3.704296] debug: unmapping init [mem 0xffffffffb3f82000-0xffffffffb4258fff] [ 3.714193] Write protecting the kernel read-only data: 28672k [ 3.717821] debug: unmapping init [mem 0xffffffffb2603000-0xffffffffb27fffff] [ 3.720359] debug: unmapping init [mem 0xffffffffb2f14000-0xffffffffb2ffffff] [ 3.759244] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.768681] systemd[1]: Detected virtualization kvm. [ 3.770628] systemd[1]: Detected architecture x86-64. [ 3.772877] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.797185] systemd[1]: No hostname configured. [ 3.799248] systemd[1]: Set hostname to . [ 3.801543] random: systemd: uninitialized urandom read (16 bytes read) [ 3.804023] systemd[1]: Initializing machine ID from random generator. [ 3.841116] random: ln: uninitialized urandom read (6 bytes read) [ 3.934906] random: systemd: uninitialized urandom read (16 bytes read) [ 3.937514] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.942433] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.946120] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Control Socket. Starting Setup Virtual Console... [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Timers. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.537825] device-mapper: uevent: version 1.0.3 [ 4.540593] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System.[ 5.189827] random: fast init done [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.236646] virtio_net virtio0 ens2: renamed from eth0 [ 5.246051] scsi host0: ata_piix [ 5.253520] scsi host1: ata_piix [ 5.255776] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.264301] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.660490] dracut-initqueue[587]: RTNETLINK answers: File exists [ 10.196260] random: crng init done [ 10.197803] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.604902] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.706801] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.993276] SELinux: Disabled at runtime. [ 12.053419] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.062742] systemd[1]: Detected virtualization kvm. [ 12.064284] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.532392] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.535576] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.540472] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.544390] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.547663] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.561914] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.569838] systemd[1]: Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Kernel Debug File System... Mounting Huge Pages File System... [ OK ] Listening on initctl Compatibility Named Pipe. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Switch Root. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice User and Session Slice. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Apply Kernel Variables... [ OK ] Stopped target Initrd File Systems. [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Remount Root and Kernel File Systems... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Mounting POSIX Message Queue File System... [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target Slices. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ 12.774633] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 13.040563] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.342393] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.357640] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.491616] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.506300] EDAC sbridge: Ver: 1.1.2 [ 15.240707] Key type dns_resolver registered [ 15.542380] NFS: Registering the id_resolver key type [ 15.544518] Key type id_resolver registered [ 15.546345] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg322-server login: [ 39.113071] hrtimer: interrupt took 2615809 ns [ 67.135249] libcfs: loading out-of-tree module taints kernel. [ 67.202312] Key type ._llcrypt registered [ 67.203876] Key type .llcrypt registered [ 67.390520] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 84.650722] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 85.877154] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 85.897356] alg: No test for adler32 (adler32-zlib) [ 87.165690] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 87.944056] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 89.711269] Key type lgssc registered [ 90.976545] Lustre: Echo OBD driver; http://www.lustre.org/ [ 104.343538] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 132.409398] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 141.242196] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 141.261300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 142.499815] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 142.529415] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 142.609799] Lustre: lustre-MDT0000: new disk, initializing [ 142.674384] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 142.698896] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 145.632135] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 154.812815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 154.910902] Lustre: 6506:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 154.938750] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 154.944367] Lustre: Skipped 1 previous similar message [ 155.019375] Lustre: lustre-MDT0001: new disk, initializing [ 155.073074] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 155.105975] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 155.118109] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 158.220449] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 162.300721] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 168.881636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 169.066453] Lustre: lustre-OST0000: new disk, initializing [ 169.069723] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 169.074719] Lustre: 8444:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 169.122945] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 171.148307] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 171.155223] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 171.228932] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 173.021332] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 183.374915] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 183.479266] Lustre: lustre-OST0001: new disk, initializing [ 183.482515] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 183.486678] Lustre: 9516:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 183.519134] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 187.646804] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 190.990433] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 191.000121] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 191.036135] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 195.795022] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 199.557829] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 203.387996] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing check_logdir /tmp/testlogs/ [ 206.572449] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing yml_node [ 209.411779] Lustre: DEBUG MARKER: Client: 2.17.55.26 [ 210.900634] Lustre: DEBUG MARKER: MDS: 2.17.55.26 [ 212.301561] Lustre: DEBUG MARKER: OSS: 2.17.55.26 [ 213.378447] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Sep 8 17:38:58 EDT 2026 [ 225.273442] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 226.329128] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 242.144436] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 242.145248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 242.152333] Lustre: Skipped 1 previous similar message [ 242.163483] Lustre: Skipped 3 previous similar messages [ 246.226918] Lustre: server umount lustre-MDT0000 complete [ 248.795891] LustreError: 6499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788903575 with bad export cookie 10963586572869375561 [ 248.796130] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 248.803205] LustreError: 6499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 252.385652] LustreError: 6514:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 252.389883] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 252.393518] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 252.404624] LustreError: 6514:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 252.414095] Lustre: Skipped 3 previous similar messages [ 253.407967] LustreError: 9970:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 253.408886] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 253.436214] LustreError: 9970:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 253.439893] Lustre: Skipped 2 previous similar messages [ 258.528274] LustreError: 6514:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 258.528278] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 258.550947] Lustre: Skipped 1 previous similar message [ 263.135870] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 263.324572] Lustre: server umount lustre-MDT0001 complete [ 266.352422] Lustre: server umount lustre-OST0000 complete [ 268.945616] Lustre: server umount lustre-OST0001 complete [ 273.269880] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 279.183579] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 301.738550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 301.909367] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 301.937677] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 302.013452] Lustre: lustre-MDT0000: new disk, initializing [ 302.065461] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 302.081182] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 305.289255] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 312.980435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 313.081491] Lustre: 15059:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 313.127616] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 313.130718] Lustre: Skipped 1 previous similar message [ 313.195138] Lustre: lustre-MDT0001: new disk, initializing [ 313.273251] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 313.316693] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 313.326465] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 315.905392] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 322.055532] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 325.837718] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 330.493524] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 330.707471] Lustre: lustre-OST0000: new disk, initializing [ 330.712480] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 330.717681] Lustre: 17027:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 330.783110] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 332.695089] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 332.703794] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 332.788734] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 334.498081] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 341.382757] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 345.013757] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 345.155242] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 348.449362] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 348.600245] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 353.249340] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 353.252820] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 353.262146] Lustre: Skipped 1 previous similar message [ 353.269455] Lustre: Skipped 1 previous similar message [ 356.108814] Lustre: server umount lustre-OST0000 complete [ 359.392866] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 359.399523] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 363.488849] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 363.489804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 363.497895] Lustre: Skipped 2 previous similar messages [ 364.714117] Lustre: server umount lustre-MDT0000 complete [ 366.713359] LustreError: 15052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788903693 with bad export cookie 10963586572869377101 [ 366.719537] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 366.719581] LustreError: 15052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 366.899830] Lustre: server umount lustre-MDT0001 complete [ 370.081932] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 17:41:35 (1788903695) [ 380.317952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 380.651862] LustreError: 19900:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 380.667585] LustreError: 19900:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 380.699285] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 382.993732] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 386.017033] LustreError: 19901:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 388.024120] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 390.655208] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 395.496213] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 399.453585] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 401.089439] Lustre: Failing over lustre-MDT0000 [ 401.307983] Lustre: server umount lustre-MDT0000 complete [ 403.936993] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 403.941963] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 403.953403] LustreError: 19901:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 403.957770] Lustre: Skipped 1 previous similar message [ 403.978197] LustreError: 19901:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 417.301048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 417.362634] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 420.215967] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 422.883672] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 422.888057] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 422.909881] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 426.565429] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 433.121665] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 433.124678] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 433.129945] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 433.136844] Lustre: Skipped 1 previous similar message [ 435.572169] Lustre: server umount lustre-MDT0000 complete [ 438.348655] LustreError: 20635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788903765 with bad export cookie 10963586572869378515 [ 438.355010] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 438.357207] LustreError: 20635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 438.639806] Lustre: server umount lustre-MDT0001 complete [ 445.089686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 445.339462] LustreError: 23233:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 445.356626] LustreError: 23233:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 445.396679] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 445.401322] Lustre: Skipped 2 previous similar messages [ 448.548796] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 454.492427] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 457.828524] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 463.194992] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 466.648389] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 467.986195] Lustre: Failing over lustre-MDT0000 [ 468.158159] Lustre: server umount lustre-MDT0000 complete [ 469.984795] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 469.993576] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 479.714279] LustreError: 23960:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 479.738176] LustreError: 23960:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 484.387113] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 484.469206] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 487.418956] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 489.956113] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 489.957171] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 489.972861] Lustre: Skipped 1 previous similar message [ 489.995816] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 493.975905] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 499.874115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 499.882110] Lustre: Skipped 4 previous similar messages [ 502.967603] Lustre: server umount lustre-MDT0000 complete [ 505.012690] LustreError: 23214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788903831 with bad export cookie 10963586572869379453 [ 505.015644] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 505.020979] LustreError: 23214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 505.171253] Lustre: server umount lustre-MDT0001 complete [ 510.075491] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 17:43:55 (1788903835) [ 519.367292] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 524.608908] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 545.258135] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 550.285928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 550.411879] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 550.438036] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 550.487141] Lustre: lustre-MDT0000: new disk, initializing [ 550.549030] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 552.544119] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 558.200426] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 558.277250] Lustre: 29594:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 558.439764] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 560.759159] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 566.042811] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 568.816330] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 572.325474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 572.472968] Lustre: 31561:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 574.188020] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 574.231663] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 574.240144] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 575.315312] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 580.464311] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 584.533897] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 594.914847] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 594.915579] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 594.927173] Lustre: Skipped 3 previous similar messages [ 594.930143] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 594.936123] LustreError: Skipped 1 previous similar message [ 594.945888] Lustre: Skipped 1 previous similar message [ 596.116453] Lustre: server umount lustre-OST0000 complete [ 612.319130] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 612.400606] Lustre: server umount lustre-MDT0000 complete [ 614.369152] LustreError: 29607:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 614.376734] LustreError: 29607:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 614.475214] LustreError: 29587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788903941 with bad export cookie 10963586572869379859 [ 614.478818] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 614.482313] LustreError: 29587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 614.699787] Lustre: server umount lustre-MDT0001 complete [ 620.473754] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 621.979097] Key type lgssc unregistered [ 622.155427] LNet: 33514:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 622.161492] LNetError: 33514:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 622.173802] LNet: Removed LNI 192.168.203.122@tcp [ 622.690140] Key type .llcrypt unregistered [ 622.693260] Key type ._llcrypt unregistered [ 635.554288] Key type ._llcrypt registered [ 635.556272] Key type .llcrypt registered [ 635.624369] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 636.091983] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 636.109426] alg: No test for adler32 (adler32-zlib) [ 637.003063] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 637.137989] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 638.743522] Key type lgssc registered [ 639.269110] Lustre: Echo OBD driver; http://www.lustre.org/ [ 643.148214] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 643.161768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 644.389250] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 646.327369] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 650.646367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 650.828660] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 652.965149] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 656.454337] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 659.206308] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 662.657162] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 662.848220] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 665.720818] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 668.846883] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 676.204954] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 687.131263] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 17:46:52 (1788904012) [ 688.151357] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 689.108970] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 692.091954] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 17:46:57 (1788904017) [ 695.960693] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 17:47:01 (1788904021) [ 696.412344] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 700.344786] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 700.349901] Lustre: Skipped 2 previous similar messages [ 701.154149] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 17:47:06 (1788904026) [ 705.655527] Lustre: Setting parameter lustre-OST0000-osc.osc.max_pages_per_rpc=1024 in log lustre-client [ 705.659261] Lustre: Skipped 4 previous similar messages [ 713.808640] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 17:47:19 (1788904039) [ 715.210242] Lustre: *** cfs_fail_loc=131b, val=2*** [ 717.112941] Lustre: *** cfs_fail_loc=131b, val=2*** [ 721.523284] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 17:47:27 (1788904047) [ 722.511840] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 722.516506] Lustre: Skipped 9 previous similar messages [ 730.286029] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 17:47:35 (1788904055) [ 737.761507] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 737.763925] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 737.769745] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 737.778515] Lustre: Skipped 1 previous similar message [ 737.784894] Lustre: Skipped 1 previous similar message [ 742.881383] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 742.886703] Lustre: Skipped 1 previous similar message [ 743.466960] Lustre: server umount lustre-OST0000 complete [ 748.002786] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 748.005704] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 748.006670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 748.018709] Lustre: Skipped 1 previous similar message [ 751.567674] Lustre: server umount lustre-MDT0000 complete [ 753.122996] LustreError: 35644:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 753.135985] LustreError: 35644:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 753.624707] LustreError: 34886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788904080 with bad export cookie 8516463014996317138 [ 753.626983] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 753.636968] LustreError: 34886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 753.804659] Lustre: server umount lustre-MDT0001 complete [ 759.220855] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 760.776058] Key type lgssc unregistered [ 760.951827] LNet: 42749:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 760.957257] LNetError: 42749:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 760.969543] LNet: Removed LNI 192.168.203.122@tcp [ 761.468995] Key type .llcrypt unregistered [ 761.470822] Key type ._llcrypt unregistered [ 771.619128] Key type ._llcrypt registered [ 771.623175] Key type .llcrypt registered [ 771.686704] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 779.963143] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 780.415155] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 780.476061] alg: No test for adler32 (adler32-zlib) [ 781.368623] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 781.484088] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 783.103547] Key type lgssc registered [ 783.622566] Lustre: Echo OBD driver; http://www.lustre.org/ [ 801.825233] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 801.840055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 802.972419] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 802.992673] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 803.044233] Lustre: lustre-MDT0000: new disk, initializing [ 803.095143] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 803.105265] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 805.407938] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 811.539095] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 811.595191] Lustre: 45877:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 811.642981] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 811.645663] Lustre: Skipped 1 previous similar message [ 811.695270] Lustre: lustre-MDT0001: new disk, initializing [ 811.745115] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 811.765366] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 811.772262] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 813.919589] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 818.895728] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 821.380966] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 825.046628] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 825.214546] Lustre: lustre-OST0000: new disk, initializing [ 825.217791] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 825.222197] Lustre: 47844:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 825.264106] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 828.685902] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 834.596671] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 834.609387] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 834.734638] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 836.480099] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 842.367761] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 842.627365] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 847.081958] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 847.284282] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 849.892641] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 849.903944] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 849.909352] Lustre: Skipped 1 previous similar message [ 855.015504] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 855.025705] Lustre: Skipped 1 previous similar message [ 860.131360] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 860.142265] Lustre: Skipped 1 previous similar message [ 863.199533] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 863.376389] Lustre: server umount lustre-OST0000 complete [ 868.321368] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 868.324820] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 868.330046] Lustre: Skipped 1 previous similar message [ 868.333329] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 870.386116] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 873.051294] Lustre: server umount lustre-MDT0000 complete [ 875.962938] LustreError: 45872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788904202 with bad export cookie 15664339231395163017 [ 875.966437] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 875.969632] LustreError: 45872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 876.207794] Lustre: server umount lustre-MDT0001 complete [ 882.902955] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 17:50:08 (1788904208) [ 887.073086] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 887.371446] LustreError: 50302:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 887.381420] LustreError: 50302:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 887.411405] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 890.179694] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 892.896348] LustreError: 50303:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 896.830557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 897.149629] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 901.595415] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 912.054721] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 918.462894] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 928.892481] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 929.360908] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 937.664849] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 947.932880] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 953.615621] LustreError: 52833:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 954.430708] Lustre: Modifying parameter general.timeout=1 in log params [ 955.747491] Lustre: Modifying parameter general.timeout=2 in log params [ 957.845291] Lustre: Modifying parameter general.timeout=4 in log params [ 957.853090] Lustre: Skipped 1 previous similar message [ 962.190712] Lustre: Modifying parameter general.timeout=8 in log params [ 962.196898] Lustre: Skipped 3 previous similar messages [ 970.881456] Lustre: Modifying parameter general.timeout=17 in log params [ 970.886904] Lustre: Skipped 8 previous similar messages [ 987.062535] Lustre: Modifying parameter general.timeout=36 in log params [ 987.066138] Lustre: Skipped 18 previous similar messages [ 1019.071427] Lustre: Modifying parameter general.timeout=73 in log params [ 1019.074044] Lustre: Skipped 36 previous similar messages [ 1074.613460] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 17:53:19 (1788904399) [ 1077.316235] Lustre: 59775:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1079.157066] Lustre: 59873:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1082.669316] Lustre: 60070:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1082.682285] Lustre: 60070:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1083.408602] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=change rule4 rate=50 in log params [ 1083.416851] Lustre: Skipped 62 previous similar messages [ 1111.864027] Lustre: 61642:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1111.872763] Lustre: 61642:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1111.897543] LustreError: 61642:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1115.768593] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 17:54:00 (1788904440) [ 1127.392565] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1127.400091] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1127.418435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1127.420939] Lustre: Skipped 2 previous similar messages [ 1133.261879] Lustre: server umount lustre-MDT0000 complete [ 1135.073758] LustreError: 51100:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1135.108138] LustreError: 51100:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1136.718801] LustreError: 50284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788904463 with bad export cookie 15664339231395163899 [ 1136.731508] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1136.734569] LustreError: 50284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1137.130383] Lustre: server umount lustre-MDT0001 complete [ 1151.098805] Lustre: server umount lustre-OST0000 complete [ 1177.622701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1177.973303] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1178.241818] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1182.700843] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1191.486677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1191.580141] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1191.620732] Lustre: 64173:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1196.452991] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1203.956907] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1209.868935] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1217.435310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1217.610127] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1217.947900] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1217.958171] Lustre: Skipped 1 previous similar message [ 1224.944717] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1234.292378] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1236.171579] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1236.183545] Lustre: Skipped 17 previous similar messages [ 1236.253867] Lustre: 66741:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1248.228137] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1248.231078] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1248.240990] LustreError: Skipped 1 previous similar message [ 1248.241474] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1248.241478] Lustre: Skipped 3 previous similar messages [ 1248.249280] Lustre: Skipped 3 previous similar messages [ 1250.557973] Lustre: server umount lustre-OST0000 complete [ 1258.464590] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1260.758444] Lustre: server umount lustre-MDT0000 complete [ 1263.586913] LustreError: 64185:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1263.606150] LustreError: 64185:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1264.542280] LustreError: 64166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788904591 with bad export cookie 15664339231395166902 [ 1264.542874] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1264.551970] LustreError: 64166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1264.963881] Lustre: server umount lustre-MDT0001 complete [ 1275.069269] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 1278.868208] Key type lgssc unregistered [ 1279.234749] LNet: 67950:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1279.243371] LNetError: 67950:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1279.256937] LNet: Removed LNI 192.168.203.122@tcp [ 1280.131493] Key type .llcrypt unregistered [ 1280.133609] Key type ._llcrypt unregistered [ 1307.853921] Key type ._llcrypt registered [ 1307.863740] Key type .llcrypt registered [ 1308.004958] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 1309.011140] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1309.022684] alg: No test for adler32 (adler32-zlib) [ 1310.147423] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 1310.357690] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 1312.111234] Key type lgssc registered [ 1313.325783] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1323.938190] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1323.965084] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1326.156683] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1332.535626] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1335.270594] LustreError: 69365:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1335.312600] LustreError: 69365:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1340.390413] LustreError: 69366:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1343.686313] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1344.456528] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1349.413858] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1356.915903] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1363.348481] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1371.625179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1372.003418] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1380.125233] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1388.871956] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1402.536084] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1415.650360] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1415.655642] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1415.676542] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1416.162463] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1416.181709] Lustre: Skipped 1 previous similar message [ 1416.188234] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1416.192159] Lustre: Skipped 2 previous similar messages [ 1417.720713] Lustre: server umount lustre-MDT0000 complete [ 1420.834574] LustreError: 72053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788904747 with bad export cookie 13180014773554178315 [ 1420.846976] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1420.850370] LustreError: 72053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1421.233868] Lustre: server umount lustre-MDT0001 complete [ 1435.165713] Lustre: server umount lustre-OST0000 complete [ 1457.231775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1457.407872] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1457.444884] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1457.693893] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1461.488732] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1469.010472] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1469.084366] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1469.105573] Lustre: 74241:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1469.538670] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1473.290858] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1479.187765] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1483.273799] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1488.117365] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1488.350176] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1488.638641] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1493.851172] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1500.259907] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1501.486885] Lustre: 76816:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1503.182882] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1509.862218] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1509.869285] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1509.881645] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1513.361172] Lustre: server umount lustre-OST0000 complete [ 1520.608549] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1520.614839] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1520.631168] LustreError: Skipped 1 previous similar message [ 1520.635738] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1520.646750] Lustre: Skipped 2 previous similar messages [ 1520.661670] Lustre: Skipped 3 previous similar messages [ 1522.986510] Lustre: server umount lustre-MDT0000 complete [ 1525.730529] LustreError: 76475:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1525.743873] LustreError: 76475:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1526.357499] LustreError: 74233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788904853 with bad export cookie 13180014773554179162 [ 1526.364717] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1526.377717] LustreError: 74233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1526.745927] Lustre: server umount lustre-MDT0001 complete [ 1536.186315] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 1538.505584] Key type lgssc unregistered [ 1538.770272] LNet: 78037:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1538.788500] LNetError: 78037:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1538.806583] LNet: Removed LNI 192.168.203.122@tcp [ 1539.440190] Key type .llcrypt unregistered [ 1539.446066] Key type ._llcrypt unregistered [ 1557.704290] Key type ._llcrypt registered [ 1557.706383] Key type .llcrypt registered [ 1557.779446] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 1558.472741] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1558.494574] alg: No test for adler32 (adler32-zlib) [ 1559.476178] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 1559.656038] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 1561.279230] Key type lgssc registered [ 1562.075261] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1567.798494] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1567.808754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1569.124298] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1572.267879] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1578.053634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1578.481498] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1581.551024] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1586.818358] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1590.823787] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1595.839629] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1596.223578] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1600.639965] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1605.843955] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1620.267125] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1620.933991] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1622.258930] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1622.267626] Lustre: Skipped 1 previous similar message [ 1624.752701] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1624.756392] Lustre: Skipped 3 previous similar messages [ 1628.826631] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1628.834674] Lustre: Skipped 5 previous similar messages [ 1637.177234] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1637.183805] Lustre: Skipped 12 previous similar messages [ 1653.536817] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1653.541449] Lustre: Skipped 26 previous similar messages [ 1686.192625] Lustre: 87531:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 1686.748119] Lustre: Disabling parameter general.jobid_var= in log params [ 1686.750436] Lustre: Skipped 46 previous similar messages [ 1687.764864] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 18:03:33 (1788905013) [ 1688.832463] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 1690.076211] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 18:03:35 (1788905015) [ 1695.712353] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1695.720424] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1695.738436] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1696.739465] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1696.740576] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1696.755838] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1698.546337] Lustre: server umount lustre-OST0000 complete [ 1701.344081] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1701.349066] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1701.361481] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1701.366023] Lustre: Skipped 1 previous similar message [ 1706.978430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1706.983841] Lustre: Skipped 2 previous similar messages [ 1712.099674] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1712.105218] Lustre: Skipped 1 previous similar message [ 1715.167197] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1715.261839] Lustre: server umount lustre-MDT0000 complete [ 1717.216566] LustreError: 80818:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1717.230673] LustreError: 80818:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1717.403634] LustreError: 87533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788905044 with bad export cookie 5742094476013233500 [ 1717.406829] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1717.412242] LustreError: 87533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1717.606255] Lustre: server umount lustre-MDT0001 complete [ 1724.101946] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 1725.856616] Key type lgssc unregistered [ 1726.024696] LNet: 88781:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1726.031691] LNetError: 88781:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1726.048668] LNet: Removed LNI 192.168.203.122@tcp [ 1726.504478] Key type .llcrypt unregistered [ 1726.506288] Key type ._llcrypt unregistered [ 1729.052661] Key type ._llcrypt registered [ 1729.054535] Key type .llcrypt registered [ 1729.126780] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 1731.975962] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules [ 1732.610949] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1732.628603] alg: No test for adler32 (adler32-zlib) [ 1733.608823] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 1733.755756] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 1735.383167] Key type lgssc registered [ 1736.152311] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1736.993197] LustreError: 89982:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 1748.669966] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 1754.163348] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1758.820311] LustreError: 89982:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 1758.852235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1760.169194] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1762.962560] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1768.498439] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 18:04:53 (1788905093) [ 1769.442378] LustreError: 91005:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1769.458522] LustreError: 91005:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1771.280987] LustreError: 91036:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1771.288673] LustreError: 91036:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1771.298347] LustreError: 91036:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 12, retries 0, failed: rc = -5 [ 1771.496546] Lustre: server umount lustre-MDT0000 complete [ 1779.095499] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 1780.718594] Key type lgssc unregistered [ 1780.879222] LNet: 92528:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1780.884122] LNetError: 92528:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1780.901070] LNet: Removed LNI 192.168.203.122@tcp [ 1781.389610] Key type .llcrypt unregistered [ 1781.392939] Key type ._llcrypt unregistered [ 1792.341866] Key type ._llcrypt registered [ 1792.343607] Key type .llcrypt registered [ 1792.401547] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 1792.870869] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1792.936848] alg: No test for adler32 (adler32-zlib) [ 1793.858501] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 1794.036797] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 1795.655488] Key type lgssc registered [ 1796.201938] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1800.479616] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1800.505151] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1801.772960] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1804.019695] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1808.610782] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1808.944287] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1811.412465] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1815.301976] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1818.066224] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1821.721964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1822.065897] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1824.993452] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1828.719072] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1860.508022] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 18:06:26 (1788905186) [ 1861.240516] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 1862.046909] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 18:06:27 (1788905187) [ 1864.163645] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1864.165068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1864.169559] Lustre: Skipped 2 previous similar messages [ 1864.172150] Lustre: Skipped 1 previous similar message [ 1869.283598] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1869.288410] Lustre: Skipped 3 previous similar messages [ 1869.474807] Lustre: server umount lustre-MDT0000 complete [ 1871.074330] LustreError: 93706:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788905197 with bad export cookie 13446392232476558779 [ 1871.077143] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1871.077809] LustreError: 93706:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1871.218239] Lustre: server umount lustre-MDT0001 complete [ 1883.278368] Lustre: server umount lustre-OST0000 complete [ 1888.357471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1888.568265] LustreError: 97990:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1888.578662] LustreError: 97990:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1888.640932] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1890.400513] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1893.856818] LustreError: 97991:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1894.262552] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1894.583329] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1896.298383] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1899.242476] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1901.436257] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1906.982086] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1907.071778] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1907.077341] LustreError: 98716:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1907.093737] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1907.098471] LustreError: 100073:0:(tgt_mount.c:2703:server_fill_super()) Unable to start targets: -98 [ 1907.101718] LustreError: 100073:0:(tgt_mount.c:2221:server_put_super()) no obd lustre-OST0000 [ 1907.104936] LustreError: 100073:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 1907.139670] Lustre: server umount lustre-OST0000 complete [ 1907.141936] LustreError: 100073:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1910.694881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1910.799450] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1910.803701] LustreError: 97978:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1910.809035] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1910.814623] LustreError: 100533:0:(tgt_mount.c:2703:server_fill_super()) Unable to start targets: -98 [ 1910.819460] LustreError: 100533:0:(tgt_mount.c:2221:server_put_super()) no obd lustre-OST0000 [ 1910.822805] LustreError: 100533:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 1910.852193] Lustre: server umount lustre-OST0000 complete [ 1910.854452] LustreError: 100533:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1914.872511] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1914.946923] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1914.949134] Lustre: Found index 0 for lustre-OST0000, updating log [ 1914.951799] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 1914.987479] Lustre: lustre-OST0000: new disk, initializing [ 1914.990105] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1914.994845] Lustre: 101069:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1915.105422] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1917.440245] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1918.977563] Lustre: Failing over lustre-OST0000 [ 1919.037165] Lustre: server umount lustre-OST0000 complete [ 1925.088420] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1925.093109] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1925.099666] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1926.680714] Lustre: server umount lustre-MDT0000 complete [ 1928.135656] LustreError: 97972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788905254 with bad export cookie 13446392232476561509 [ 1928.139347] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1928.140844] LustreError: 97972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1928.278690] Lustre: server umount lustre-MDT0001 complete [ 1931.645791] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 18:07:37 (1788905257) [ 1937.574366] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 1941.591839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1941.719593] LustreError: 103411:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1941.740765] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1943.163583] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1946.231278] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1947.933816] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1949.032178] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 1951.512189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1953.828505] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1957.071642] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1957.105208] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 1957.140516] Lustre: lustre-OST0001: new disk, initializing [ 1957.142520] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1957.145442] Lustre: 105727:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1959.553938] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1961.986921] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1961.991675] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1962.036091] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1963.121788] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1968.994170] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1971.041894] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.22@tcp (stopping) [ 1971.045258] Lustre: Skipped 1 previous similar message [ 1972.195619] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1972.196290] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 1972.200616] Lustre: Skipped 2 previous similar messages [ 1976.154775] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.22@tcp (stopping) [ 1976.158359] Lustre: Skipped 3 previous similar messages [ 1976.534475] Lustre: server umount lustre-MDT0001 complete [ 1977.313585] LustreError: 103412:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1977.320549] LustreError: 103412:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1979.861725] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1979.922466] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1979.924366] Lustre: Found index 1 for lustre-MDT0001, updating log [ 1979.940350] Lustre: 103399:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1980.225168] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1980.229962] Lustre: Skipped 3 previous similar messages [ 1981.698056] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1981.797401] LustreError: 107507:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 1981.800550] LustreError: 107507:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 1981.804101] LustreError: 103403:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 1981.829561] LustreError: 107513:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.122@tcp: cfg command failed: rc = -17 [ 1981.833092] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 1981.833092] [ 1985.056034] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 18:08:30 (1788905310) [ 1985.510488] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1991.017315] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 1996.904437] Lustre: 109336:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2005.024046] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2006.171802] Lustre: 110470:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2052.064151] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2052.064401] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2052.064795] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2052.071591] Lustre: Skipped 4 previous similar messages [ 2057.631900] Lustre: server umount lustre-MDT0000 complete [ 2058.826599] LustreError: 103391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788905385 with bad export cookie 13446392232476562433 [ 2058.827925] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2058.830213] LustreError: 103391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2058.926305] Lustre: server umount lustre-MDT0001 complete [ 2069.507474] Lustre: server umount lustre-OST0000 complete [ 2080.996379] Lustre: server umount lustre-OST0001 complete [ 2082.895529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2085.905796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2088.563025] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2091.521576] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2097.522509] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 2100.931792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2100.940117] Lustre: lustre-MDT0000: reset Object Index mappings [ 2101.079615] LustreError: 114392:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2101.102209] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2102.446483] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2105.241661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2105.249679] Lustre: lustre-MDT0001: reset Object Index mappings [ 2106.970447] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2107.942560] Lustre: 115539:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2110.123138] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2111.394954] LustreError: 115892:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2111.397931] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 2112.209459] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2115.015503] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2116.200739] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 2116.202321] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 2116.232235] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 2116.939946] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2120.035956] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2121.217223] Lustre: 117415:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2135.510910] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 18:11:01 (1788905461) [ 2166.752067] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2166.756062] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2166.761727] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2166.764050] Lustre: Skipped 7 previous similar messages [ 2171.103643] Lustre: server umount lustre-MDT0000 complete [ 2172.298242] LustreError: 114375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788905499 with bad export cookie 13446392232476609207 [ 2172.302620] LustreError: 114375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2172.303085] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2172.896031] LustreError: 116083:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2172.903111] LustreError: 116083:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2178.546889] Lustre: server umount lustre-MDT0001 complete [ 2185.926306] Lustre: server umount lustre-OST0000 complete [ 2193.300215] Lustre: server umount lustre-OST0001 complete [ 2195.413414] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 2198.640700] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 2210.927484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2211.011065] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2211.021397] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2211.057718] Lustre: lustre-MDT0000: new disk, initializing [ 2211.083038] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2212.414229] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2215.332058] Lustre: server umount lustre-MDT0000 complete [ 2218.451984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2218.549825] Lustre: Found index 0 for lustre-MDT0000, updating log [ 2218.554627] Lustre: 121673:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2218.559562] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 2220.013637] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2221.687835] Lustre: server umount lustre-MDT0000 complete [ 2224.407700] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 18:12:30 (1788905550) [ 2224.897171] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 2225.480767] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 18:12:31 (1788905551) [ 2230.705841] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 2233.817341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2233.934397] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2233.936622] Lustre: Skipped 5 previous similar messages [ 2235.207105] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2238.405755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2238.454316] Lustre: 123724:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2238.484276] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2238.487668] Lustre: Skipped 1 previous similar message [ 2238.542850] Lustre: lustre-MDT0001: new disk, initializing [ 2238.611093] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2238.617252] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2240.434703] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2243.749954] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 2248.090398] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2248.255411] Lustre: lustre-OST0000: new disk, initializing [ 2248.258068] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2248.262122] Lustre: 125299:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2249.835108] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2249.840868] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2249.950077] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2252.239866] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2259.151944] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2259.259291] Lustre: lustre-OST0001: new disk, initializing [ 2259.261592] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2259.265340] Lustre: 126169:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2261.056059] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2261.083197] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2261.233474] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2264.634805] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2273.474254] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2276.773918] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2519.177913] Lustre: 132559:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 2519.184059] Lustre: 132559:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 2519.187581] Lustre: 132559:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 2519.191347] Lustre: 132559:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 2519.196645] Lustre: 132559:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 2519.203471] Lustre: 132559:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 2729.305928] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 18:20:54 (1788906054) [ 2742.754157] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2742.758477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2742.765221] Lustre: Skipped 6 previous similar messages [ 2742.771654] Lustre: Skipped 9 previous similar messages [ 2748.754385] Lustre: server umount lustre-MDT0000 complete [ 2751.677317] LustreError: 123716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788906078 with bad export cookie 13446392232476662260 [ 2751.679822] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2751.685861] LustreError: 123716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2752.993548] LustreError: 126971:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2753.005962] LustreError: 126971:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2757.982647] Lustre: server umount lustre-MDT0001 complete [ 2766.696507] Lustre: server umount lustre-OST0000 complete [ 2783.199115] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2783.334136] Lustre: server umount lustre-OST0001 complete [ 2787.797543] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 2794.150946] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 2816.529238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2816.711230] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2816.738543] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2816.789343] Lustre: lustre-MDT0000: new disk, initializing [ 2816.830588] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2816.835233] Lustre: Skipped 3 previous similar messages [ 2816.845766] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2819.346691] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2826.249501] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2826.330525] Lustre: 135595:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2826.539103] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2829.263244] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2835.438291] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2838.721312] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2843.512308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2843.714876] Lustre: 137565:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2845.225854] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2845.232868] Lustre: Skipped 1 previous similar message [ 2845.237468] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2845.336500] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2847.981242] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2854.587623] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2860.314529] Lustre: lustre-MDD0000: changelog on [ 2861.939838] Lustre: lustre-MDD0001: changelog on [ 4344.060817] Lustre: lustre-MDD0001: changelog off [ 4344.901619] Lustre: 137697:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 4345.284223] Lustre: lustre-MDD0000: changelog off [ 4350.432355] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4350.435883] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4350.441951] Lustre: Skipped 4 previous similar messages [ 4350.445510] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4350.449207] Lustre: Skipped 9 previous similar messages [ 4350.944645] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4353.330262] Lustre: server umount lustre-OST0000 complete [ 4356.064753] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4356.068960] Lustre: Skipped 2 previous similar messages [ 4361.186124] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4361.189332] Lustre: Skipped 4 previous similar messages [ 4369.375264] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4369.492703] Lustre: server umount lustre-MDT0000 complete [ 4370.401305] LustreError: 135609:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4370.409493] LustreError: 135609:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4370.949455] LustreError: 145688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788907697 with bad export cookie 13446392232477426989 [ 4370.953078] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4370.956248] LustreError: 145688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4371.114355] Lustre: server umount lustre-MDT0001 complete [ 4375.676771] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 4376.931693] Key type lgssc unregistered [ 4377.078592] LNet: 146640:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4377.082846] LNetError: 146640:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4377.091545] LNet: Removed LNI 192.168.203.122@tcp [ 4377.484171] Key type .llcrypt unregistered [ 4377.485979] Key type ._llcrypt unregistered [ 4385.014568] Key type ._llcrypt registered [ 4385.016096] Key type .llcrypt registered [ 4385.061845] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 4392.988870] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 4393.380898] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4393.450842] alg: No test for adler32 (adler32-zlib) [ 4394.322935] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 4394.440723] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 4396.039345] Key type lgssc registered [ 4396.497804] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4410.194137] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4410.201309] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4411.289786] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4411.302608] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4411.340516] Lustre: lustre-MDT0000: new disk, initializing [ 4411.383565] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4411.391430] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4413.021961] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4417.647416] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4417.696755] Lustre: 149814:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4417.727265] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4417.730204] Lustre: Skipped 1 previous similar message [ 4417.769870] Lustre: lustre-MDT0001: new disk, initializing [ 4417.811131] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4417.826153] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4417.830449] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4419.308594] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4423.426228] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4425.462879] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4428.175535] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4428.294340] Lustre: lustre-OST0000: new disk, initializing [ 4428.296874] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4428.300666] Lustre: 151782:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4428.340521] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4430.537901] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4434.951203] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4434.955917] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4434.966766] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4435.009646] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4437.517355] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_module obdecho/obdecho [ 4438.490753] LustreError: 152784:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 4438.495174] LustreError: 152784:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 4439.448803] LustreError: 152880:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 4445.154338] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4445.155885] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4445.165665] Lustre: Skipped 2 previous similar messages [ 4445.173324] Lustre: Skipped 2 previous similar messages [ 4450.275470] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4450.278369] Lustre: Skipped 2 previous similar messages [ 4450.599596] Lustre: server umount lustre-MDT0000 complete [ 4452.160279] LustreError: 151781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788907778 with bad export cookie 11157612568101723392 [ 4452.164644] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4452.166281] LustreError: 151781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4452.305246] Lustre: server umount lustre-MDT0001 complete [ 4464.230672] Lustre: server umount lustre-OST0000 complete [ 4473.099497] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 4477.448320] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 4493.609816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4493.705556] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4493.724639] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4493.770246] Lustre: lustre-MDT0000: new disk, initializing [ 4493.813969] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4493.823467] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4495.529276] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4500.325617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4500.376031] Lustre: 156589:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4500.415699] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4500.418400] Lustre: Skipped 1 previous similar message [ 4500.458681] Lustre: lustre-MDT0001: new disk, initializing [ 4500.501879] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4500.526864] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4500.537736] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4502.291335] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4506.636707] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4508.922591] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4511.846660] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4511.964808] Lustre: lustre-OST0000: new disk, initializing [ 4511.967244] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4511.970894] Lustre: 158553:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4512.011922] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4512.150190] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4512.155191] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4512.199229] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4514.768202] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4519.198654] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4521.136581] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4521.233407] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4523.120914] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4523.216584] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4527.584932] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4527.585722] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4527.595721] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4527.600663] Lustre: Skipped 1 previous similar message [ 4530.266242] Lustre: server umount lustre-OST0000 complete [ 4532.704461] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4532.705799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4532.710398] Lustre: Skipped 2 previous similar messages [ 4537.824219] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4537.826914] Lustre: Skipped 2 previous similar messages [ 4538.209873] Lustre: server umount lustre-MDT0000 complete [ 4539.628107] LustreError: 156581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788907866 with bad export cookie 11157612568101724337 [ 4539.630467] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4539.632365] LustreError: 156581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4539.755680] Lustre: server umount lustre-MDT0001 complete [ 4541.146229] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 18:51:06 (1788907866) [ 4541.781233] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 4542.454227] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 18:51:08 (1788907868) [ 4545.348254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4545.559619] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4547.151045] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4550.317343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4552.099111] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4555.309498] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4557.712523] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4560.597620] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4563.074871] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4566.326261] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4576.226764] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4576.227271] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4576.228014] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4576.233393] Lustre: Skipped 2 previous similar messages [ 4576.236969] LustreError: Skipped 1 previous similar message [ 4580.453907] Lustre: server umount lustre-MDT0000 complete [ 4581.345679] LustreError: 161687:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4581.352453] LustreError: 161687:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 4581.832196] LustreError: 160941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788907908 with bad export cookie 11157612568101725205 [ 4581.834560] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4581.838211] LustreError: 160941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4582.233878] LustreError: 160956:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4586.463778] LustreError: 160956:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4586.463962] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4591.584415] LustreError: 162796:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4591.590681] LustreError: 162796:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4592.474243] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.22@tcp (stopping) [ 4592.477805] Lustre: Skipped 7 previous similar messages [ 4596.191125] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4596.322290] Lustre: server umount lustre-MDT0001 complete [ 4598.859657] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4604.429620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4604.565433] LustreError: 164799:0:(llog.c:1655:llog_backup()) MGC192.168.203.122@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 4604.569516] Lustre: 164799:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.122@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 4617.186397] LustreError: 164804:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4617.193609] LustreError: 164804:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4626.400374] LustreError: 164823:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4626.406883] LustreError: 164823:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 4642.650472] LustreError: 164804:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4642.659561] LustreError: 164804:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 4665.823384] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 4665.828395] Lustre: 164799:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.122@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 4677.088281] LustreError: 164804:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4677.096175] LustreError: 164804:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 21 previous similar messages [ 4706.793786] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4706.799424] Lustre: Skipped 2 previous similar messages [ 4708.426393] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4711.672493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4711.913926] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4726.239814] LustreError: 148040:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000004abd8670 x1875805702558592/t0(0) o250->MGC192.168.203.122@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4727.899865] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4730.747176] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4731.368557] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4731.407359] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4733.546795] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4735.214444] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 1 sec [ 4746.317556] Lustre: server umount lustre-MDT0000 complete [ 4746.721644] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4746.722433] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4746.722637] LustreError: 162786:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4746.722644] LustreError: 162786:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 35 previous similar messages [ 4746.729070] Lustre: Skipped 1 previous similar message [ 4747.848315] LustreError: 162799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788908074 with bad export cookie 11157612568101731652 [ 4747.851383] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4747.854921] LustreError: 162799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4747.982747] Lustre: server umount lustre-MDT0001 complete [ 4759.773932] Lustre: server umount lustre-OST0000 complete [ 4766.633300] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 4770.357504] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 4784.410283] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4784.490697] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4784.502237] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4784.540699] Lustre: lustre-MDT0000: new disk, initializing [ 4784.571749] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4784.575252] Lustre: Skipped 1 previous similar message [ 4784.584970] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4786.085342] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4790.629370] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4790.682336] Lustre: 169872:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4790.799917] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4792.345421] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4796.673205] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4798.869686] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4801.598421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4801.710500] Lustre: 171844:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4802.236171] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4802.297723] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4803.998442] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4808.200916] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4809.982364] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4810.069631] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4811.687922] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4811.769170] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4812.768258] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4812.769214] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4812.772367] LustreError: Skipped 1 previous similar message [ 4812.779015] Lustre: Skipped 1 previous similar message [ 4812.779946] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4812.786053] Lustre: Skipped 1 previous similar message [ 4818.651816] Lustre: server umount lustre-OST0000 complete [ 4821.472296] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4826.457150] Lustre: server umount lustre-MDT0000 complete [ 4827.898166] LustreError: 169867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788908154 with bad export cookie 11157612568101748480 [ 4827.900441] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4827.905305] LustreError: 169867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4828.029913] Lustre: server umount lustre-MDT0001 complete [ 4831.656542] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 18:55:57 (1788908157) [ 4834.137895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4835.951134] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4839.545462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4841.307043] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4844.422189] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4846.626440] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4849.616198] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4852.253771] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4855.388485] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4865.362288] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 18:56:31 (1788908191) [ 4868.065453] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4868.070875] Lustre: Skipped 4 previous similar messages [ 4878.304982] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4878.307733] Lustre: Skipped 10 previous similar messages [ 4880.863178] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4880.918997] Lustre: server umount lustre-OST0000 complete [ 4888.475369] Lustre: server umount lustre-MDT0000 complete [ 4888.544416] LustreError: 174252:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4888.550807] LustreError: 174252:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 4889.927918] LustreError: 174234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788908216 with bad export cookie 11157612568101749348 [ 4889.929772] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4889.933402] LustreError: 174234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4890.072933] Lustre: server umount lustre-MDT0001 complete [ 4894.392918] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 4895.500968] Key type lgssc unregistered [ 4895.649797] LNet: 177977:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4895.653382] LNetError: 177977:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4895.667497] LNet: Removed LNI 192.168.203.122@tcp [ 4896.033307] Key type .llcrypt unregistered [ 4896.034358] Key type ._llcrypt unregistered [ 4904.744743] Key type ._llcrypt registered [ 4904.746823] Key type .llcrypt registered [ 4904.796264] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 4905.187825] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4905.214560] alg: No test for adler32 (adler32-zlib) [ 4906.106861] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 4906.209145] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 4907.815204] Key type lgssc registered [ 4908.225835] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4911.386102] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4911.396335] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4926.879358] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4931.999257] LustreError: 179198:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.122@tcp: failed processing log, type 4: rc = -110 [ 4937.119442] LustreError: 179198:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 4937.124403] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 4937.126072] LustreError: MGC192.168.203.122@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 4937.131249] LustreError: 179198:0:(tgt_mount.c:1966:server_start_targets()) failed to start server lustre-OST0000: -22 [ 4937.135411] LustreError: 179198:0:(tgt_mount.c:2703:server_fill_super()) Unable to start targets: -22 [ 4937.138307] LustreError: 179198:0:(tgt_mount.c:2221:server_put_super()) no obd lustre-OST0000 [ 4937.158465] Lustre: server umount lustre-OST0000 complete [ 4937.160180] LustreError: 179198:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 4940.213115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4941.374419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4942.781630] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4945.814070] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4945.941392] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4947.384756] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4949.887741] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4951.866800] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4954.335348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4954.415275] LustreError: 181510:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 4954.419305] Lustre: 181510:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.203.122@tcp: invalid local config log lustre-OST0000: rc = -22 [ 4954.472482] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4956.546403] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4959.348411] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4960.738832] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4960.740564] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4960.744045] Lustre: Skipped 2 previous similar messages [ 4960.746240] Lustre: Skipped 1 previous similar message [ 4965.857230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4965.860225] Lustre: Skipped 3 previous similar messages [ 4966.298277] Lustre: server umount lustre-MDT0000 complete [ 4967.510113] LustreError: 179676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788908294 with bad export cookie 8892490515879363113 [ 4967.512257] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4967.514377] LustreError: 179676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4967.613838] Lustre: server umount lustre-MDT0001 complete [ 4979.241919] Lustre: server umount lustre-OST0000 complete [ 4982.777131] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 4987.034971] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 5001.290314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5001.382119] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5001.396920] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5001.436708] Lustre: lustre-MDT0000: new disk, initializing [ 5001.470226] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5001.477853] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5003.011890] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5007.489614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5007.534485] Lustre: 185030:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5007.560603] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5007.563051] Lustre: Skipped 1 previous similar message [ 5007.600285] Lustre: lustre-MDT0001: new disk, initializing [ 5007.634584] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5007.648370] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5007.652727] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5009.146822] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5013.206319] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5015.169972] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5018.001941] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5018.126793] Lustre: lustre-OST0000: new disk, initializing [ 5018.129084] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5018.132179] Lustre: 186996:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5018.172713] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5019.323237] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5019.328043] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5019.359240] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5020.415444] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5024.584351] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5026.109180] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5026.186151] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5027.585286] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5027.669480] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5029.857976] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5029.860377] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5029.860864] LustreError: Skipped 1 previous similar message [ 5029.867528] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5029.871012] Lustre: Skipped 1 previous similar message [ 5034.455155] Lustre: server umount lustre-OST0000 complete [ 5038.559896] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5038.563426] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5038.568996] Lustre: Skipped 1 previous similar message [ 5038.572187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5045.216490] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5045.219336] Lustre: Skipped 3 previous similar messages [ 5050.335220] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5050.401276] Lustre: server umount lustre-MDT0000 complete [ 5051.721351] LustreError: 185023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788908378 with bad export cookie 8892490515879363813 [ 5051.724643] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5051.726201] LustreError: 185023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5051.845860] Lustre: server umount lustre-MDT0001 complete [ 5054.875955] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 18:59:40 (1788908380) [ 5060.846734] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 5061.834526] Key type lgssc unregistered [ 5061.975351] LNet: 189939:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5061.977488] LNetError: 189939:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5061.984443] LNet: Removed LNI 192.168.203.122@tcp [ 5062.288879] Key type .llcrypt unregistered [ 5062.290552] Key type ._llcrypt unregistered [ 5069.661256] Key type ._llcrypt registered [ 5069.662633] Key type .llcrypt registered [ 5069.697751] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 5070.126086] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5070.138218] alg: No test for adler32 (adler32-zlib) [ 5070.987751] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 5071.076902] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 5072.647191] Key type lgssc registered [ 5073.025046] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5075.470162] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5075.476779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5076.625020] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5077.935731] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5080.823541] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5080.949932] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5082.437651] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5085.043296] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5086.919949] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5089.528539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5089.600915] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5089.603154] LustreError: 192996:0:(llog.c:1696:llog_backup()) MGC192.168.203.122@tcp: failed to backup log lustre-OST0000: rc = -28 [ 5089.606953] Lustre: 192996:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.203.122@tcp: can't backup local config lustre-OST0000: rc = -28 [ 5089.646041] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5091.557588] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5093.926603] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5095.905609] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5095.906773] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5095.910019] Lustre: Skipped 2 previous similar messages [ 5095.911760] Lustre: Skipped 1 previous similar message [ 5100.752899] Lustre: server umount lustre-MDT0000 complete [ 5101.025685] LustreError: 191133:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5101.034396] LustreError: 191133:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 5101.971050] LustreError: 191115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788908428 with bad export cookie 5873260767383110534 [ 5101.974331] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5101.975783] LustreError: 191115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5102.089640] Lustre: server umount lustre-MDT0001 complete [ 5113.691800] Lustre: server umount lustre-OST0000 complete [ 5116.665568] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 5119.896540] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 5132.306412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5132.396720] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5132.407539] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5132.444800] Lustre: lustre-MDT0000: new disk, initializing [ 5132.467051] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5132.472829] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5133.915439] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5138.153251] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5138.196228] Lustre: 196513:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5138.216926] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5138.218635] Lustre: Skipped 1 previous similar message [ 5138.220627] LustreError: 197309:0:(llog.c:1696:llog_backup()) MGC192.168.203.122@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 5138.224127] LustreError: 197309:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 5138.226784] Lustre: 197309:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.122@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 5138.234971] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5138.237023] Lustre: Skipped 1 previous similar message [ 5138.270171] Lustre: lustre-MDT0001: new disk, initializing [ 5138.303792] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5138.315175] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5138.318964] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5139.601935] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5143.221858] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5144.987781] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5147.459471] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5147.554170] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5147.555593] Lustre: Skipped 1 previous similar message [ 5147.556875] LustreError: 198470:0:(llog.c:1696:llog_backup()) MGC192.168.203.122@tcp: failed to backup log lustre-OST0000: rc = -28 [ 5147.560606] LustreError: 198470:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 5147.564071] Lustre: 198470:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.122@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 5147.567700] Lustre: 198470:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 5147.577276] Lustre: lustre-OST0000: new disk, initializing [ 5147.579324] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5147.582797] Lustre: 198494:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5147.624359] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5149.309284] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5149.312867] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5149.349408] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5149.723500] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5153.608473] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5155.227344] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5155.325885] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5156.937979] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5157.037265] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5159.908792] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5159.909454] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5159.911873] LustreError: Skipped 1 previous similar message [ 5159.912859] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5159.918053] Lustre: Skipped 1 previous similar message [ 5159.919960] Lustre: Skipped 1 previous similar message [ 5165.027898] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5165.030946] Lustre: Skipped 2 previous similar messages [ 5170.145356] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5170.148090] Lustre: Skipped 1 previous similar message [ 5172.191112] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5172.243283] Lustre: server umount lustre-OST0000 complete [ 5174.239875] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5174.245620] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5174.250554] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5179.808578] Lustre: server umount lustre-MDT0000 complete [ 5180.384023] LustreError: 196527:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5180.390297] LustreError: 196527:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5181.098459] LustreError: 196508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788908507 with bad export cookie 5873260767383111220 [ 5181.100631] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5181.102737] LustreError: 196508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5181.209875] Lustre: server umount lustre-MDT0001 complete [ 5184.130686] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 19:01:50 (1788908510) [ 5189.979733] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 5193.850284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5193.993951] LustreError: 201614:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5194.009832] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5195.588778] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5199.030839] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5200.764626] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5201.864432] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5204.482409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5206.848722] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5210.027253] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5210.104736] Lustre: lustre-OST0001: new disk, initializing [ 5210.107289] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5210.110870] Lustre: 203919:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5210.323957] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5210.328705] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5210.373139] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5212.398225] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5218.403357] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5225.544960] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5226.335377] Lustre: DEBUG MARKER: ADD OST3 [ 5228.354290] loop: module loaded [ 5228.379572] loop0: detected capacity change from 0 to 204800000 [ 5228.513144] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5228.517609] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5228.522614] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5230.943225] loop0: detected capacity change from 0 to 204800000 [ 5233.305883] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5233.401217] Lustre: lustre-OST0003: new disk, initializing [ 5233.404031] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 5233.409716] Lustre: 205659:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5233.467620] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 5233.471777] Lustre: Skipped 3 previous similar messages [ 5233.731282] Lustre: DEBUG MARKER: STOP OST3 [ 5234.581616] Lustre: Failing over lustre-OST0003 [ 5234.629034] Lustre: server umount lustre-OST0003 complete [ 5235.192126] LustreError: 203963:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5235.209237] LustreError: 203963:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5236.434437] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5240.295379] LustreError: 203963:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5240.310386] LustreError: 203963:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5241.455371] Lustre: DEBUG MARKER: START OST3 again [ 5248.217698] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5253.645606] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 5253.655039] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 5253.697456] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000401 [ 5254.391528] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5273.136381] Lustre: Failing over lustre-OST0003 [ 5273.211862] Lustre: server umount lustre-OST0003 complete [ 5274.084056] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5274.088118] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5274.098573] Lustre: Skipped 2 previous similar messages [ 5274.103018] LustreError: 203963:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5274.111197] LustreError: 203963:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 5291.493193] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5291.510629] LustreError: Skipped 1 previous similar message [ 5291.525079] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5291.559527] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5291.568233] Lustre: Skipped 2 previous similar messages [ 5294.572159] LustreError: 203099:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5294.590303] LustreError: 203099:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 5305.311198] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5305.441626] Lustre: server umount lustre-MDT0000 complete [ 5307.504617] LustreError: 201595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788908634 with bad export cookie 5873260767383112088 [ 5307.506512] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5307.512714] LustreError: 201595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5309.921944] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5309.925450] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5309.933623] Lustre: Skipped 4 previous similar messages [ 5309.943902] Lustre: Skipped 13 previous similar messages [ 5313.784420] Lustre: server umount lustre-MDT0001 complete [ 5316.481746] Lustre: server umount lustre-OST0000 complete [ 5318.904130] Lustre: server umount lustre-OST0001 complete [ 5323.010517] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 5328.234413] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 5347.389568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5347.536525] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5347.556619] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5347.609583] Lustre: lustre-MDT0000: new disk, initializing [ 5347.659644] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5347.663104] Lustre: Skipped 1 previous similar message [ 5347.676245] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5350.115481] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5356.585272] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5356.653158] Lustre: 211070:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5356.852745] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5359.334367] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5365.030545] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5368.329440] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5372.385218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5372.531727] Lustre: lustre-OST0000: new disk, initializing [ 5372.534518] Lustre: Skipped 1 previous similar message [ 5372.539276] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5372.545822] Lustre: Skipped 2 previous similar messages [ 5372.549222] Lustre: 213036:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5374.312105] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5374.317748] Lustre: Skipped 1 previous similar message [ 5374.326493] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5374.391390] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5376.325349] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5382.174695] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5384.665244] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5384.780315] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5386.635276] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5386.731060] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5389.794895] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5389.798367] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5389.804573] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5389.804953] Lustre: Skipped 2 previous similar messages [ 5389.808569] Lustre: Skipped 1 previous similar message [ 5402.079116] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5402.164060] Lustre: server umount lustre-OST0000 complete [ 5410.664740] Lustre: server umount lustre-MDT0000 complete [ 5413.008115] LustreError: 211864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788908739 with bad export cookie 5873260767383114321 [ 5413.009618] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5413.036931] LustreError: 211864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5413.290450] Lustre: server umount lustre-MDT0001 complete [ 5415.661260] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 19:05:41 (1788908741) [ 5425.446531] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 5430.987828] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 5453.640136] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5453.781951] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5453.808923] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5453.869406] Lustre: lustre-MDT0000: new disk, initializing [ 5453.924067] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5456.871557] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5464.069128] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5464.155512] Lustre: 217639:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5464.337658] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5466.964396] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5472.960148] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5476.293810] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5480.454042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5480.614830] Lustre: 219607:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5480.691928] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5480.695911] Lustre: Skipped 4 previous similar messages [ 5482.021061] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5484.408734] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5490.429757] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5493.154211] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5493.280946] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5495.831971] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5495.982304] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5497.315227] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5497.316842] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5497.317715] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5497.317720] Lustre: Skipped 9 previous similar messages [ 5497.323200] Lustre: Skipped 3 previous similar messages [ 5497.328703] LustreError: Skipped 1 previous similar message [ 5503.251220] Lustre: server umount lustre-OST0000 complete [ 5511.860261] Lustre: server umount lustre-MDT0000 complete [ 5512.672857] LustreError: 218447:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5512.680027] LustreError: 218447:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 5514.132450] LustreError: 217632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788908840 with bad export cookie 5873260767383115203 [ 5514.133557] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5514.140591] LustreError: 217632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5514.327746] Lustre: server umount lustre-MDT0001 complete [ 5519.434164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5522.046636] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5527.075348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5529.718135] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5534.712454] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5538.711497] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5543.096079] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5547.163196] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5551.895490] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5562.851838] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5562.853121] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5562.877716] Lustre: Skipped 2 previous similar messages [ 5565.915082] Lustre: server umount lustre-OST0000 complete [ 5582.303135] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5582.410111] Lustre: server umount lustre-MDT0000 complete [ 5584.647198] LustreError: 221804:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788908911 with bad export cookie 5873260767383116085 [ 5584.650948] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5584.653207] LustreError: 221804:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5584.870686] Lustre: server umount lustre-MDT0001 complete [ 5686.569029] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 5688.181865] Key type lgssc unregistered [ 5688.389769] LNet: 225482:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5688.393961] LNetError: 225482:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5688.411690] LNet: Removed LNI 192.168.203.122@tcp [ 5688.935165] Key type .llcrypt unregistered [ 5688.938996] Key type ._llcrypt unregistered [ 5704.983543] Key type ._llcrypt registered [ 5704.984777] Key type .llcrypt registered [ 5705.044861] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 5713.766597] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 5714.146239] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5714.199666] alg: No test for adler32 (adler32-zlib) [ 5715.098963] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 5715.215935] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 5716.823153] Key type lgssc registered [ 5717.431748] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5735.349648] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5735.364238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5736.492214] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5736.511595] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5736.555390] Lustre: lustre-MDT0000: new disk, initializing [ 5736.611894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5736.625824] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5738.704465] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5744.351039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5744.402075] Lustre: 228807:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5744.455404] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5744.458014] Lustre: Skipped 1 previous similar message [ 5744.494749] Lustre: lustre-MDT0001: new disk, initializing [ 5744.540172] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5744.556952] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5744.561908] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5746.208941] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5750.819652] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5753.177064] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5756.145896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5756.279961] Lustre: lustre-OST0000: new disk, initializing [ 5756.282775] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5756.286746] Lustre: 230778:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5756.341866] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5759.069915] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5764.017983] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5765.822787] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5766.155294] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5766.161822] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5766.202819] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5766.943375] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 5768.806420] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5768.911959] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5771.235447] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5771.235971] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5771.240125] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5771.244825] Lustre: Skipped 1 previous similar message [ 5771.951094] Lustre: server umount lustre-OST0000 complete [ 5775.328551] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5775.333620] LustreError: Skipped 1 previous similar message [ 5775.336695] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5775.345967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5775.349923] Lustre: Skipped 1 previous similar message [ 5776.353378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5776.353465] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5781.472115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5781.475029] Lustre: Skipped 2 previous similar messages [ 5786.593173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5786.597028] Lustre: Skipped 1 previous similar message [ 5788.127273] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5788.226076] Lustre: server umount lustre-MDT0000 complete [ 5790.074419] LustreError: 229604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788909116 with bad export cookie 7781458955656062907 [ 5790.077518] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5790.079273] LustreError: 229604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5790.264906] Lustre: server umount lustre-MDT0001 complete [ 5799.197178] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 5803.768512] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5803.975586] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5805.794545] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5809.121520] LustreError: 233713:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5809.129635] LustreError: 233713:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5809.824778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5810.017605] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5812.129295] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5813.655405] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5816.956358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5819.843534] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5823.750529] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5823.845761] Lustre: lustre-OST0001: new disk, initializing [ 5823.849358] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5823.854329] Lustre: 236019:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5823.903286] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5823.905886] Lustre: Skipped 1 previous similar message [ 5826.791666] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5831.182550] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5831.187161] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5831.231284] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5833.306689] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5835.548859] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5841.669691] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 19:12:47 (1788909167) [ 5844.816985] Lustre: server umount lustre-MDT0000 complete [ 5845.984140] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5845.991469] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5846.001055] LustreError: 233713:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5846.498390] LustreError: 236018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788909173 with bad export cookie 7781458955656063775 [ 5846.505141] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5846.663149] Lustre: server umount lustre-MDT0001 complete [ 5858.837218] Lustre: server umount lustre-OST0000 complete [ 5870.720671] Lustre: server umount lustre-OST0001 complete [ 5873.780963] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 5878.500875] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 5895.200182] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5895.304843] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5895.323244] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5895.362741] Lustre: lustre-MDT0000: new disk, initializing [ 5895.404313] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5895.413346] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5897.225096] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5902.216310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5902.274299] Lustre: 239962:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5902.386362] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5903.941148] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5908.099368] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5910.435477] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5912.972320] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5913.081402] Lustre: lustre-OST0000: new disk, initializing [ 5913.083367] Lustre: Skipped 1 previous similar message [ 5913.085494] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5913.088246] Lustre: Skipped 2 previous similar messages [ 5913.091076] Lustre: 241930:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5914.492606] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5914.499083] Lustre: Skipped 1 previous similar message [ 5914.502067] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5914.541471] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5915.560359] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5919.911923] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5921.637412] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5921.722742] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5923.436441] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5923.525923] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5924.835401] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5924.836219] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5924.838809] LustreError: Skipped 1 previous similar message [ 5924.843914] Lustre: Skipped 2 previous similar messages [ 5924.846982] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5924.850572] Lustre: Skipped 1 previous similar message [ 5930.392211] Lustre: server umount lustre-OST0000 complete [ 5933.024549] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5933.028249] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5945.313506] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5945.316393] Lustre: Skipped 8 previous similar messages [ 5946.335152] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5946.410962] Lustre: server umount lustre-MDT0000 complete [ 5948.200394] LustreError: 240756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788909275 with bad export cookie 7781458955656065882 [ 5948.208156] LustreError: 240756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5948.208300] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5948.392048] Lustre: server umount lustre-MDT0001 complete [ 5952.284445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5952.488065] LustreError: 244143:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5952.500627] LustreError: 244143:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5952.528720] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5952.531947] Lustre: Skipped 2 previous similar messages [ 5954.514907] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5957.603055] LustreError: 244144:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5958.758530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5960.835549] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5964.323657] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5966.974178] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5970.371403] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5973.306951] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5977.250240] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6121.442045] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6121.447693] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6121.456714] Lustre: Skipped 1 previous similar message [ 6121.460036] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6126.677555] Lustre: server umount lustre-OST0000 complete [ 6134.657292] Lustre: server umount lustre-MDT0000 complete [ 6136.648757] LustreError: 244878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788909463 with bad export cookie 7781458955656066764 [ 6136.650915] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6136.656417] LustreError: 244878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6136.819335] Lustre: server umount lustre-MDT0001 complete [ 6242.312747] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 6243.567358] Key type lgssc unregistered [ 6243.723524] LNet: 247746:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6243.728392] LNetError: 247746:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6243.741552] LNet: Removed LNI 192.168.203.122@tcp [ 6244.139212] Key type .llcrypt unregistered [ 6244.141210] Key type ._llcrypt unregistered [ 6254.115570] Key type ._llcrypt registered [ 6254.117113] Key type .llcrypt registered [ 6254.162249] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 6261.218619] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 6261.551926] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6261.567233] alg: No test for adler32 (adler32-zlib) [ 6262.450286] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 6262.557235] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 6264.151151] Key type lgssc registered [ 6264.641194] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6285.060125] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 6289.451273] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6289.462385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6290.567489] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6290.585163] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6290.627630] Lustre: lustre-MDT0000: new disk, initializing [ 6290.667241] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6290.678385] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6292.546264] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6297.865414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6297.930474] Lustre: 251789:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6297.964864] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6297.967765] Lustre: Skipped 1 previous similar message [ 6298.010820] Lustre: lustre-MDT0001: new disk, initializing [ 6298.063893] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6298.089267] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6298.094198] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6300.002766] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6302.634310] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6305.609233] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6305.743549] Lustre: lustre-OST0000: new disk, initializing [ 6305.746223] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6305.749920] Lustre: 253420:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6305.796612] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6308.403676] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6311.436413] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6311.442916] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6311.494245] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6313.865088] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6313.962577] Lustre: lustre-OST0001: new disk, initializing [ 6313.966896] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6313.970792] Lustre: 254288:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6314.027715] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6316.703024] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6321.691089] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6321.696198] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6321.741337] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6324.392767] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6329.904503] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6336.995149] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6336.998869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6337.002785] Lustre: Skipped 2 previous similar messages [ 6337.008131] Lustre: Skipped 3 previous similar messages [ 6337.287654] Lustre: server umount lustre-MDT0000 complete [ 6339.185401] LustreError: 251779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788909665 with bad export cookie 3404055520758185486 [ 6339.188584] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6339.191848] LustreError: 251779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6341.402244] Lustre: server umount lustre-MDT0001 complete [ 6352.414853] Lustre: server umount lustre-OST0000 complete [ 6364.549464] Lustre: server umount lustre-OST0001 complete [ 6371.370781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6376.038353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6380.264196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6385.274029] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6394.367143] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 6399.281120] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6399.295536] Lustre: lustre-MDT0000: reset Object Index mappings [ 6399.559392] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6401.697680] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6404.577978] LustreError: 259293:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6404.587316] LustreError: 259293:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6405.668490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6405.678886] Lustre: lustre-MDT0001: reset Object Index mappings [ 6407.755522] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6409.246325] Lustre: 260434:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6412.548854] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6412.723829] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6412.726522] Lustre: Skipped 1 previous similar message [ 6415.438220] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6415.783131] LustreError: 260787:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6418.967219] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6421.546275] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6424.235558] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 6424.246176] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6430.904484] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6432.411787] Lustre: 262304:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6434.787661] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6434.791816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6434.794279] Lustre: Skipped 3 previous similar messages [ 6434.798555] Lustre: Skipped 3 previous similar messages [ 6439.907626] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6439.912508] Lustre: Skipped 2 previous similar messages [ 6440.112347] Lustre: server umount lustre-MDT0000 complete [ 6441.965618] LustreError: 260037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788909768 with bad export cookie 3404055520758192276 [ 6441.967334] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6441.973275] LustreError: 260037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6442.166443] Lustre: server umount lustre-MDT0001 complete [ 6453.262453] Lustre: server umount lustre-OST0000 complete [ 6465.223337] Lustre: server umount lustre-OST0001 complete [ 6473.069820] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 19:23:18 (1788909798) [ 6479.968892] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 6484.229349] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 6500.486737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6500.611928] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6500.634870] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6500.675115] Lustre: lustre-MDT0000: new disk, initializing [ 6500.707730] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6500.710674] Lustre: Skipped 1 previous similar message [ 6500.719485] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6502.570986] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6508.213522] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6508.290884] Lustre: 266551:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6508.470303] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6510.822205] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6516.549127] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6519.496448] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6523.378429] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6523.529481] Lustre: lustre-OST0000: new disk, initializing [ 6523.532037] Lustre: Skipped 1 previous similar message [ 6523.535538] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6523.539699] Lustre: Skipped 2 previous similar messages [ 6523.544302] Lustre: 268522:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6524.821837] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6524.827657] Lustre: Skipped 1 previous similar message [ 6524.831690] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6524.888259] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6526.450420] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6530.997684] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6532.946313] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6533.062616] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6535.120970] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6535.221646] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6540.262075] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6540.262955] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6540.263489] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6540.268319] Lustre: Skipped 2 previous similar messages [ 6542.317584] Lustre: server umount lustre-OST0000 complete [ 6544.355359] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6544.358956] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6544.368367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6544.370721] Lustre: Skipped 1 previous similar message [ 6550.130744] Lustre: server umount lustre-MDT0000 complete [ 6550.496675] LustreError: 266564:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6550.503527] LustreError: 266564:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6551.742579] LustreError: 268521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788909878 with bad export cookie 3404055520758194362 [ 6551.743882] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6551.749462] LustreError: 268521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6551.906544] Lustre: server umount lustre-MDT0001 complete [ 6559.573362] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 6563.859054] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6564.042845] LustreError: 271451:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6564.069355] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6564.072373] Lustre: Skipped 2 previous similar messages [ 6565.639205] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6569.079683] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6570.936902] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6572.108281] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6574.581244] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6577.191407] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6581.062710] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6581.145720] Lustre: lustre-OST0001: new disk, initializing [ 6581.148642] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6581.152128] Lustre: 273758:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6582.865274] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6582.870162] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6582.915393] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6583.724552] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6589.019683] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6590.587933] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6592.936280] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 6592.946655] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 6599.651275] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6599.653827] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6599.658176] Lustre: Skipped 3 previous similar messages [ 6599.662579] Lustre: Skipped 5 previous similar messages [ 6601.657183] Lustre: server umount lustre-MDT0000 complete [ 6603.141306] LustreError: 271432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788909929 with bad export cookie 3404055520758195244 [ 6603.145388] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6603.147066] LustreError: 271432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6603.281915] Lustre: server umount lustre-MDT0001 complete [ 6615.050688] Lustre: server umount lustre-OST0000 complete [ 6626.866563] Lustre: server umount lustre-OST0001 complete [ 6634.075121] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 6638.862345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6639.076996] LustreError: 276810:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6639.109361] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6639.114021] Lustre: Skipped 3 previous similar messages [ 6641.078544] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6644.848850] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6646.699202] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6647.905778] Lustre: 277951:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6650.671710] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6653.672583] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6654.948338] LustreError: 278304:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6654.954773] LustreError: 278304:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6657.564893] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6660.207382] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6664.181059] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 6664.183632] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 6669.354408] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6670.833526] Lustre: 279828:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6672.851704] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 6672.886497] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 6676.574830] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 19:26:42 (1788910002) [ 6689.763703] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6689.768450] Lustre: Skipped 4 previous similar messages [ 6689.768602] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6689.772625] Lustre: Skipped 3 previous similar messages [ 6690.334319] Lustre: server umount lustre-MDT0000 complete [ 6691.627406] LustreError: 277554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788910018 with bad export cookie 3404055520758201551 [ 6691.630806] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6691.632379] LustreError: 277554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6694.880053] LustreError: 276806:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6694.885631] LustreError: 276806:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6697.775942] Lustre: server umount lustre-MDT0001 complete [ 6713.311273] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6713.361427] Lustre: server umount lustre-OST0000 complete [ 6720.793416] Lustre: server umount lustre-OST0001 complete [ 6723.104652] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 6726.760966] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 6743.163334] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 6746.918179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6747.016070] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6747.030583] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6747.068874] Lustre: lustre-MDT0000: new disk, initializing [ 6747.103962] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6748.666997] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6753.907607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6753.981941] Lustre: 284057:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6754.137870] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6755.867422] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6760.804746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6760.913220] Lustre: 285692:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6761.894954] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6763.049901] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6767.396849] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6767.499280] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6767.502286] Lustre: Skipped 6 previous similar messages [ 6768.734214] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6769.565715] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6774.183700] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6780.571063] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 19:28:26 (1788910106) [ 6783.970710] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6783.973782] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6783.975109] Lustre: Skipped 4 previous similar messages [ 6783.978305] Lustre: Skipped 7 previous similar messages [ 6787.497231] Lustre: server umount lustre-MDT0000 complete [ 6789.044253] LustreError: 284853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788910115 with bad export cookie 3404055520758209447 [ 6789.047975] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6789.048262] LustreError: 284853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6789.087914] LustreError: 284066:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6789.095249] LustreError: 284066:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6789.182709] Lustre: server umount lustre-MDT0001 complete [ 6796.742461] Lustre: server umount lustre-OST0000 complete [ 6798.197397] Lustre: server umount lustre-OST0001 complete [ 6801.268767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6803.000992] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6806.417341] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6807.982273] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6810.599377] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6812.494859] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6814.931329] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6817.148335] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6819.667460] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6836.589217] loop1: detected capacity change from 0 to 204800000 [ 6836.661982] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6836.666428] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6836.670791] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6838.425784] loop1: detected capacity change from 0 to 204800000 [ 6840.412475] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6840.443430] LustreError: 288794:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs9325: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 6840.450190] LustreError: 288794:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs9325 [ 6840.452812] LustreError: 288794:0:(mgs_handler.c:517:mgs_target_reg()) fs9325-MDT0001: Error checking target index: rc = -13 [ 6840.457073] LustreError: fs9325-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 6840.461753] LustreError: 292538:0:(tgt_mount.c:2703:server_fill_super()) Unable to start targets: -13 [ 6840.465778] LustreError: 292538:0:(tgt_mount.c:2221:server_put_super()) no obd fs9325-MDT0001 [ 6840.468923] LustreError: 292538:0:(tgt_mount.c:116:server_deregister_mount()) fs9325-MDT0001 not registered [ 6840.481695] Lustre: server umount fs9325-MDT0001 complete [ 6840.483651] LustreError: 292538:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 6843.487485] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6843.517945] Lustre: Setting parameter fs9325-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs9325-MDT0001 [ 6843.521099] Lustre: Skipped 2 previous similar messages [ 6843.538445] Lustre: srv-fs9325-MDT0001: No data found on store. Initialize space. [ 6843.541322] Lustre: Skipped 4 previous similar messages [ 6843.579651] Lustre: fs9325-MDT0001: new disk, initializing [ 6843.581415] Lustre: Skipped 3 previous similar messages [ 6843.615238] Lustre: srv-fs9325-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 6843.618460] Lustre: 293003:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs9325-MDT0001-mdtlov: can't create new llog: rc = -115 [ 6843.621323] LustreError: 293003:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs9325-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 6845.079131] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6848.147208] Lustre: server umount fs9325-MDT0001 complete [ 6852.835827] loop1: detected capacity change from 0 to 204800000 [ 6853.094392] LustreError: 290635:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6853.106640] LustreError: 290635:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 19 previous similar messages [ 6855.290319] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6855.422503] Lustre: 294569:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs9325-MDT0001-mdtlov: can't create new llog: rc = -115 [ 6855.427909] LustreError: 294569:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs9325-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 6857.069591] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6858.969301] Lustre: server umount fs9325-MDT0001 complete [ 6862.815861] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6862.819366] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6862.823531] Lustre: Skipped 3 previous similar messages [ 6862.825933] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6862.828751] Lustre: Skipped 3 previous similar messages [ 6868.459055] Lustre: server umount lustre-MDT0000 complete [ 6871.516804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6871.559054] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6873.023761] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6875.525135] loop1: detected capacity change from 0 to 204800000 [ 6877.159938] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6877.432744] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6877.552655] Lustre: 297011:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs9325-MDT0001-mdtlov: can't create new llog: rc = -115 [ 6877.556279] LustreError: 297011:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs9325-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 6879.011777] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6880.749210] Lustre: server umount fs9325-MDT0001 complete [ 6888.927770] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6893.582361] Lustre: server umount lustre-OST0000 complete [ 6897.633124] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6897.636271] LustreError: Skipped 1 previous similar message [ 6909.407093] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6909.470613] Lustre: server umount lustre-MDT0000 complete [ 6910.675926] LustreError: 288835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788910237 with bad export cookie 3404055520758212121 [ 6910.677069] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6910.680730] LustreError: 288835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6910.801366] Lustre: server umount lustre-MDT0001 complete [ 6914.740091] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 6915.814575] Key type lgssc unregistered [ 6915.947501] LNet: 299353:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6915.951213] LNetError: 299353:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6915.961303] LNet: Removed LNI 192.168.203.122@tcp [ 6916.281114] Key type .llcrypt unregistered [ 6916.282755] Key type ._llcrypt unregistered [ 6924.664115] Key type ._llcrypt registered [ 6924.665705] Key type .llcrypt registered [ 6924.717597] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 6925.348779] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6925.362103] alg: No test for adler32 (adler32-zlib) [ 6926.254280] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 6926.372438] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 6927.967218] Key type lgssc registered [ 6928.432687] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6931.332492] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6931.340946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6932.520138] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6934.108182] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6937.369524] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6937.549970] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6939.219181] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6941.861451] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6943.945664] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6946.425990] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6946.575824] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6947.624181] LustreError: 302426:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6947.632158] LustreError: 302426:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6948.806081] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6951.462465] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6952.932251] LustreError: 302427:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6958.051889] LustreError: 302602:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6958.061392] LustreError: 302602:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6962.019238] LustreError: 303097:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6967.130806] LustreError: 302426:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6967.138638] LustreError: 302426:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6968.824412] loop1: detected capacity change from 0 to 204800000 [ 6968.944220] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6968.948930] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6968.954463] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6970.886515] loop1: detected capacity change from 0 to 204800000 [ 6973.009338] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6973.072609] Lustre: fs16597-OST0000: new disk, initializing [ 6973.075546] Lustre: srv-fs16597-OST0000: No data found on store. Initialize space. [ 6973.078838] Lustre: 304350:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6973.105584] Lustre: fs16597-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6975.456797] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6977.371243] LustreError: 302428:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6977.378607] LustreError: 302428:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6979.352639] Lustre: server umount fs16597-OST0000 complete [ 6984.076242] loop1: detected capacity change from 0 to 204800000 [ 6986.170482] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6986.253928] Lustre: fs16597-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6988.435557] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6991.629200] loop2: detected capacity change from 0 to 204800000 [ 6991.703820] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6991.708832] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6991.714856] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6993.800436] loop2: detected capacity change from 0 to 204800000 [ 6993.896566] LustreError: 302428:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6993.905828] LustreError: 302428:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 6996.085883] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6996.123377] LustreError: fs16597-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [ 6996.130948] LustreError: fs16597-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 6996.135962] LustreError: 307385:0:(tgt_mount.c:2703:server_fill_super()) Unable to start targets: -13 [ 6996.139181] LustreError: 307385:0:(tgt_mount.c:2221:server_put_super()) no obd fs16597-OST0001 [ 6996.142293] LustreError: 307385:0:(tgt_mount.c:116:server_deregister_mount()) fs16597-OST0001 not registered [ 6996.157972] Lustre: server umount fs16597-OST0001 complete [ 6996.160521] LustreError: 307385:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 7000.312302] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7000.367927] Lustre: fs16597-OST0001: new disk, initializing [ 7000.370923] Lustre: srv-fs16597-OST0001: No data found on store. Initialize space. [ 7000.374304] Lustre: 307933:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7000.404491] Lustre: fs16597-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7002.700972] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7006.569963] Lustre: server umount fs16597-OST0000 complete [ 7010.084168] Lustre: server umount fs16597-OST0001 complete [ 7014.672409] loop1: detected capacity change from 0 to 204800000 [ 7016.932531] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7017.036346] Lustre: fs16597-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7019.526509] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7022.065476] loop2: detected capacity change from 0 to 204800000 [ 7024.126605] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7026.258970] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7028.569528] LustreError: 302428:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7028.576702] LustreError: 302428:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 18 previous similar messages [ 7029.497900] Lustre: server umount fs16597-OST0000 complete [ 7032.970768] Lustre: server umount fs16597-OST0001 complete [ 7039.974170] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7039.974729] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7039.976769] LustreError: Skipped 2 previous similar messages [ 7039.977637] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7039.980159] Lustre: Skipped 1 previous similar message [ 7045.090422] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7045.091540] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [ 7045.093299] Lustre: Skipped 2 previous similar messages [ 7050.214173] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7050.216669] Lustre: Skipped 1 previous similar message [ 7051.231112] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7051.300258] Lustre: server umount lustre-OST0000 complete [ 7055.329360] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7055.329505] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7055.334528] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7055.339668] Lustre: Skipped 1 previous similar message [ 7059.036989] Lustre: server umount lustre-MDT0000 complete [ 7060.322365] LustreError: 301341:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788910387 with bad export cookie 11613543930147370842 [ 7060.325855] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7060.326787] LustreError: 301341:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7060.443749] Lustre: server umount lustre-MDT0001 complete [ 7064.616914] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 7065.787808] Key type lgssc unregistered [ 7065.932619] LNet: 313845:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7065.936409] LNetError: 313845:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7065.949315] LNet: Removed LNI 192.168.203.122@tcp [ 7066.305157] Key type .llcrypt unregistered [ 7066.307921] Key type ._llcrypt unregistered [ 7077.687275] Key type ._llcrypt registered [ 7077.688619] Key type .llcrypt registered [ 7077.731622] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 7084.409814] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 7084.867138] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7084.873989] alg: No test for adler32 (adler32-zlib) [ 7085.731934] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 7085.829425] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 7087.415142] Key type lgssc registered [ 7087.804410] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7100.586649] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7100.603500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7101.702241] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7101.717137] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7101.760922] Lustre: lustre-MDT0000: new disk, initializing [ 7101.799262] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7101.807312] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7103.324916] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7107.997985] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7108.043647] Lustre: 317676:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7108.069282] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7108.071676] Lustre: Skipped 1 previous similar message [ 7108.105245] Lustre: lustre-MDT0001: new disk, initializing [ 7108.137860] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7108.152461] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7108.157065] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7109.665050] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7113.781159] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7115.873967] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7118.461034] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7118.587447] Lustre: lustre-OST0000: new disk, initializing [ 7118.590336] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7118.594380] Lustre: 319644:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7118.636318] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7120.870310] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7125.093681] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7126.142502] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7126.146468] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7126.194052] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7131.633505] Lustre: Mounted lustre-client [ 7132.046537] Lustre: Unmounted lustre-client [ 7136.738381] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7136.738482] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7136.742457] LustreError: Skipped 1 previous similar message [ 7136.750442] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7136.754902] Lustre: Skipped 1 previous similar message [ 7141.793280] Lustre: server umount lustre-OST0000 complete [ 7143.904326] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7143.907355] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7143.911397] Lustre: Skipped 1 previous similar message [ 7143.914036] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7146.976411] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7146.977183] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7149.340813] Lustre: server umount lustre-MDT0000 complete [ 7150.758234] LustreError: 318471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788910477 with bad export cookie 14121695273424517714 [ 7150.761553] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7150.763078] LustreError: 318471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7150.896542] Lustre: server umount lustre-MDT0001 complete [ 7154.791793] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 7155.920056] Key type lgssc unregistered [ 7178.591491] LNet: 321801:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7178.595050] LNetError: 321801:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7255.397310] LNet: Removed LNI 192.168.203.122@tcp [ 7255.688674] Key type .llcrypt unregistered [ 7255.689843] Key type ._llcrypt unregistered [ 7262.014717] Key type ._llcrypt registered [ 7262.015808] Key type .llcrypt registered [ 7262.054021] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 7268.751786] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 7269.264385] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7269.318655] alg: No test for adler32 (adler32-zlib) [ 7270.195376] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 7270.298308] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 7271.887196] Key type lgssc registered [ 7272.372871] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7284.400935] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 19:36:50 (1788910610) [ 7287.264416] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7287.279219] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7288.397773] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7288.411545] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7288.451729] Lustre: lustre-MDT0000: new disk, initializing [ 7288.482510] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7288.489873] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7290.195142] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7295.045346] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7295.089250] Lustre: 325000:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7295.113160] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7295.115273] Lustre: Skipped 1 previous similar message [ 7295.151550] Lustre: lustre-MDT0001: new disk, initializing [ 7295.190170] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7295.202814] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7295.207156] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7296.785322] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7301.056928] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7303.061511] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7305.463560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7305.592521] Lustre: lustre-OST0000: new disk, initializing [ 7305.594969] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7305.598830] Lustre: 326965:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7305.630319] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7307.786621] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7308.358579] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7308.363219] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7308.395777] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7311.884593] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7319.009982] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7319.012508] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7319.013917] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7319.018497] Lustre: Skipped 1 previous similar message [ 7320.149582] Lustre: server umount lustre-OST0000 complete [ 7323.676145] Lustre: server umount lustre-MDT0000 complete [ 7324.129286] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7324.134397] Lustre: Skipped 1 previous similar message [ 7324.955533] LustreError: 324991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788910651 with bad export cookie 7754907969516590056 [ 7324.960054] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7324.960884] LustreError: 324991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7325.077220] Lustre: server umount lustre-MDT0001 complete [ 7329.060444] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 7330.135138] Key type lgssc unregistered [ 7330.266380] LNet: 328866:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7330.269032] LNetError: 328866:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7330.275270] LNet: Removed LNI 192.168.203.122@tcp [ 7330.621090] Key type .llcrypt unregistered [ 7330.622532] Key type ._llcrypt unregistered [ 7339.549028] Key type ._llcrypt registered [ 7339.550374] Key type .llcrypt registered [ 7339.591493] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 7340.004880] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7340.016586] alg: No test for adler32 (adler32-zlib) [ 7340.901820] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 7341.009126] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 7342.591185] Key type lgssc registered [ 7343.000747] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7345.834825] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7345.843042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7346.996789] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7348.379755] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7351.620050] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7351.770147] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7353.370614] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7356.172124] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7358.363707] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7361.189964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7361.388291] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7363.708249] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7366.513964] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7375.656278] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7376.276785] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7381.986318] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7381.988055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7381.994191] Lustre: Skipped 2 previous similar messages [ 7381.997637] Lustre: Skipped 1 previous similar message [ 7387.104746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7387.107355] Lustre: Skipped 4 previous similar messages [ 7392.223135] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7392.228708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7392.234647] Lustre: Skipped 3 previous similar messages [ 7392.344185] Lustre: server umount lustre-MDT0000 complete [ 7392.602738] LustreError: 330114:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7392.611768] LustreError: 330114:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7394.137118] LustreError: 330098:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788910720 with bad export cookie 7541400131004917726 [ 7394.142648] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7394.145833] LustreError: 330098:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7397.343858] LustreError: 330112:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7397.345826] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7397.363331] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7399.392621] LustreError: 330112:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7399.400294] LustreError: 330112:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7400.449807] Lustre: server umount lustre-MDT0001 complete [ 7404.112825] Lustre: server umount lustre-OST0000 complete [ 7410.831885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7414.495733] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7416.706331] Lustre: 334298:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.122@tcp [ 7418.082829] Lustre: server umount MGS complete [ 7424.810801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7425.111475] LustreError: 334868:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7425.165712] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7428.578724] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7430.625098] LustreError: 334869:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7434.893962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7435.125907] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7438.751945] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7446.898534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7447.248252] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7455.089808] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7463.357387] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7464.930528] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [ 7470.223324] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7471.594342] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7477.735799] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 7477.743927] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7477.760477] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7477.770857] Lustre: Skipped 2 previous similar messages [ 7478.579496] Lustre: server umount lustre-OST0000 complete [ 7485.413859] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7485.432792] Lustre: Skipped 1 previous similar message [ 7489.806189] Lustre: server umount lustre-MDT0000 complete [ 7490.528445] LustreError: 334869:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7490.549661] LustreError: 334869:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7492.749491] LustreError: 334851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788910819 with bad export cookie 7541400131004918552 [ 7492.766140] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7502.426502] Lustre: server umount lustre-MDT0001 complete [ 7508.257912] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 7510.014662] Key type lgssc unregistered [ 7510.209839] LNet: 338305:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7510.217397] LNetError: 338305:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7510.234373] LNet: Removed LNI 192.168.203.122@tcp [ 7510.722706] Key type .llcrypt unregistered [ 7510.724558] Key type ._llcrypt unregistered [ 7524.220407] Key type ._llcrypt registered [ 7524.222381] Key type .llcrypt registered [ 7524.293126] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 7535.579928] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 7536.157126] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7536.201758] alg: No test for adler32 (adler32-zlib) [ 7537.202548] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 7537.376764] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 7539.007194] Key type lgssc registered [ 7539.764542] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7563.163867] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7563.175897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7564.319856] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7564.348713] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7564.402643] Lustre: lustre-MDT0000: new disk, initializing [ 7564.458738] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7564.471964] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7567.039520] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7573.241802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7573.295596] Lustre: 341444:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7573.327122] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7573.330158] Lustre: Skipped 1 previous similar message [ 7573.369845] Lustre: lustre-MDT0001: new disk, initializing [ 7573.433291] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7573.460448] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7573.469250] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7575.729730] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7581.248340] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7584.497544] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7588.442068] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7588.603526] Lustre: lustre-OST0000: new disk, initializing [ 7588.607129] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7588.612571] Lustre: 343414:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7588.670936] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7592.300413] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7597.613413] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7597.620901] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7597.697813] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7598.519169] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7601.418994] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7601.558952] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7604.194924] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7604.311377] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7607.776557] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7607.785806] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7607.795534] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7608.292872] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7608.296478] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7608.308064] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7608.311338] Lustre: Skipped 1 previous similar message [ 7611.734187] Lustre: server umount lustre-OST0000 complete [ 7614.432463] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7614.438240] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7614.450512] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7618.532997] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7618.535804] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7620.625536] Lustre: server umount lustre-MDT0000 complete [ 7623.251698] LustreError: 343413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788910950 with bad export cookie 7880066826468649703 [ 7623.255792] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7623.261942] LustreError: 343413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7623.463382] Lustre: server umount lustre-MDT0001 complete [ 7629.148777] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 19:42:34 (1788910954) [ 7640.108553] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 7645.791792] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 7666.580695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7666.771926] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7666.789187] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7666.831803] Lustre: lustre-MDT0000: new disk, initializing [ 7666.876233] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7666.897318] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7669.616197] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7676.528570] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7676.596353] Lustre: 348164:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7676.633374] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7676.636665] Lustre: Skipped 1 previous similar message [ 7676.683891] Lustre: lustre-MDT0001: new disk, initializing [ 7676.743901] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7676.780675] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7676.789349] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7679.323613] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7684.606995] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7687.069688] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7690.178308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7690.303185] Lustre: lustre-OST0000: new disk, initializing [ 7690.305788] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7690.309424] Lustre: 350131:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7690.350285] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7692.944656] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7695.879516] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7695.885370] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7695.929134] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7697.435679] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7716.320355] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7716.321442] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7716.326884] LustreError: Skipped 1 previous similar message [ 7716.327602] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7716.334658] Lustre: Skipped 1 previous similar message [ 7716.344356] Lustre: Skipped 2 previous similar messages [ 7717.610917] Lustre: server umount lustre-OST0000 complete [ 7720.929782] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 7725.435265] Lustre: server umount lustre-MDT0000 complete [ 7727.170371] LustreError: 348200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788911053 with bad export cookie 7880066826468650585 [ 7727.173226] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7727.177901] LustreError: 348200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7727.323636] Lustre: server umount lustre-MDT0001 complete [ 7732.063468] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 7733.412872] Key type lgssc unregistered [ 7733.569365] LNet: 352279:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7733.576269] LNetError: 352279:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7733.583349] LNet: Removed LNI 192.168.203.122@tcp [ 7733.975375] Key type .llcrypt unregistered [ 7733.977170] Key type ._llcrypt unregistered [ 7774.104759] Key type ._llcrypt registered [ 7774.106299] Key type .llcrypt registered [ 7774.158376] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 7774.300240] Key type .llcrypt unregistered [ 7774.301829] Key type ._llcrypt unregistered [ 7808.423167] Key type ._llcrypt registered [ 7808.425776] Key type .llcrypt registered [ 7808.482869] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 7808.620841] Key type .llcrypt unregistered [ 7808.623237] Key type ._llcrypt unregistered [ 7822.486713] Key type ._llcrypt registered [ 7822.488277] Key type .llcrypt registered [ 7822.532261] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 7822.665761] Key type .llcrypt unregistered [ 7822.667117] Key type ._llcrypt unregistered [ 7836.250962] Key type ._llcrypt registered [ 7836.252718] Key type .llcrypt registered [ 7836.295260] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 7836.427452] Key type .llcrypt unregistered [ 7836.428814] Key type ._llcrypt unregistered [ 7869.932149] Key type ._llcrypt registered [ 7869.933693] Key type .llcrypt registered [ 7869.974155] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 7870.084255] Key type .llcrypt unregistered [ 7870.085969] Key type ._llcrypt unregistered [ 7881.882741] Key type ._llcrypt registered [ 7881.884290] Key type .llcrypt registered [ 7881.922173] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 7882.023314] Key type .llcrypt unregistered [ 7882.024619] Key type ._llcrypt unregistered [ 7893.265056] Key type ._llcrypt registered [ 7893.266466] Key type .llcrypt registered [ 7893.313082] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 7893.426497] Key type .llcrypt unregistered [ 7893.428394] Key type ._llcrypt unregistered [ 7904.768623] Key type ._llcrypt registered [ 7904.769547] Key type .llcrypt registered [ 7904.802583] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 7904.900469] Key type .llcrypt unregistered [ 7904.901772] Key type ._llcrypt unregistered [ 7915.930153] Key type ._llcrypt registered [ 7915.931180] Key type .llcrypt registered [ 7915.967715] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 7916.062461] Key type .llcrypt unregistered [ 7916.063538] Key type ._llcrypt unregistered [ 7927.185592] Key type ._llcrypt registered [ 7927.187805] Key type .llcrypt registered [ 7927.225747] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 7927.329449] Key type .llcrypt unregistered [ 7927.331052] Key type ._llcrypt unregistered [ 7947.683938] Key type ._llcrypt registered [ 7947.686178] Key type .llcrypt registered [ 7947.718991] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 7947.808165] Key type .llcrypt unregistered [ 7947.809161] Key type ._llcrypt unregistered [ 7958.849585] Key type ._llcrypt registered [ 7958.850847] Key type .llcrypt registered [ 7958.893189] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 7959.008237] Key type .llcrypt unregistered [ 7959.009748] Key type ._llcrypt unregistered [ 7969.841541] Key type ._llcrypt registered [ 7969.843039] Key type .llcrypt registered [ 7969.887866] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 7969.991328] Key type .llcrypt unregistered [ 7969.992616] Key type ._llcrypt unregistered [ 7987.866546] Key type ._llcrypt registered [ 7987.867617] Key type .llcrypt registered [ 7987.903463] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 7994.143893] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 7994.602426] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7994.615596] alg: No test for adler32 (adler32-zlib) [ 7995.463480] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 7995.547819] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 7997.127139] Key type lgssc registered [ 7997.474199] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8008.161363] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8008.166492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8009.240397] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [ 8009.249681] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [ 8009.282636] Lustre: tst250fs-MDT0000: new disk, initializing [ 8009.301872] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8009.308732] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8010.608773] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8014.562993] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8014.604379] Lustre: 372185:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8014.626369] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [ 8014.628211] Lustre: Skipped 1 previous similar message [ 8014.660806] Lustre: tst250fs-MDT0001: new disk, initializing [ 8014.689485] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8014.700736] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8014.703747] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8015.936504] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8018.156119] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8020.337315] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8020.444344] Lustre: tst250fs-OST0000: new disk, initializing [ 8020.447337] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [ 8020.450268] Lustre: 373814:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8020.481739] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8022.341901] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8026.107916] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8026.113078] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8026.155940] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8026.474943] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8026.539121] Lustre: tst250fs-OST0001: new disk, initializing [ 8026.541255] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [ 8026.543379] Lustre: 374687:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8026.571377] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8028.551245] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8031.016511] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [ 8033.137249] Lustre: 372189:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.122@tcp: IR log tst250fs-mdtir failed, not fatal: rc = -2 [ 8041.952993] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8041.954317] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 8041.958433] Lustre: Skipped 2 previous similar messages [ 8041.960377] Lustre: Skipped 1 previous similar message [ 8047.073071] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 8047.075855] Lustre: Skipped 5 previous similar messages [ 8047.192624] Lustre: server umount tst250fs-MDT0000 complete [ 8048.448200] LustreError: 372980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788911375 with bad export cookie 17627166588655191735 [ 8048.450560] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8048.453720] LustreError: 372980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8048.565160] Lustre: server umount tst250fs-MDT0001 complete [ 8059.393874] Lustre: server umount tst250fs-OST0000 complete [ 8070.872432] Lustre: server umount tst250fs-OST0001 complete [ 8076.914651] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 8079.936842] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 8090.633574] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 19:50:16 (1788911416) [ 8091.173691] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [ 8096.649813] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 8100.484752] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 8110.900057] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7897 sec ========== 19:50:36 (1788911436) [ 8111.470913] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 19:50:37 (1788911437) === [ 8112.557618] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 19:50:38 (1788911438) === [ 8124.826960] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 8125.776968] Key type lgssc unregistered [ 8125.888227] LNet: 383785:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8125.891152] LNetError: 383785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8125.903293] LNet: Removed LNI 192.168.203.122@tcp [ 8126.184113] Key type .llcrypt unregistered [ 8126.185369] Key type ._llcrypt unregistered