[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 470661335 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001015] APIC: Switch to symmetric I/O mode setup [ 0.003133] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009015] pid_max: default: 32768 minimum: 301 [ 0.010181] LSM: Security Framework initializing [ 0.011064] Yama: becoming mindful. [ 0.012043] SELinux: Initializing. [ 0.013088] *** VALIDATE selinux *** [ 0.021179] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026522] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027155] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028118] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029116] *** VALIDATE tmpfs *** [ 0.030410] *** VALIDATE proc *** [ 0.031266] *** VALIDATE cgroup *** [ 0.032010] *** VALIDATE cgroup2 *** [ 0.033283] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034158] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036005] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037026] Spectre V2 : User space: Vulnerable [ 0.038007] Speculative Store Bypass: Vulnerable [ 0.041624] debug: unmapping init [mem 0xffffffff89059000-0xffffffff89060fff] [ 0.044000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044797] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045029] ... version: 2 [ 0.046016] ... bit width: 48 [ 0.047013] ... generic registers: 4 [ 0.048014] ... value mask: 0000ffffffffffff [ 0.049015] ... max period: 00007fffffffffff [ 0.050015] ... fixed-purpose events: 3 [ 0.051013] ... event mask: 000000070000000f [ 0.052324] rcu: Hierarchical SRCU implementation. [ 0.054547] smp: Bringing up secondary CPUs ... [ 0.055638] x86: Booting SMP configuration: [ 0.056030] .... node #0, CPUs: #1 #2 #3 [ 0.059323] smp: Brought up 1 node, 4 CPUs [ 0.061016] smpboot: Max logical packages: 1 [ 0.062019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.144737] node 0 deferred pages initialised in 80ms [ 0.148173] devtmpfs: initialized [ 0.149224] x86/mm: Memory block size: 128MB [ 0.151644] gcov: version magic: 0x41383552 [ 0.153358] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.154090] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.155243] pinctrl core: initialized pinctrl subsystem [ 0.156217] [ 0.156593] ************************************************************* [ 0.157011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.158012] ** ** [ 0.159013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.160012] ** ** [ 0.161010] ** This means that this kernel is built to expose internal ** [ 0.162011] ** IOMMU data structures, which may compromise security on ** [ 0.163012] ** your system. ** [ 0.164012] ** ** [ 0.165014] ** If you see this message and you are not debugging the ** [ 0.166009] ** kernel, report this immediately to your vendor! ** [ 0.167011] ** ** [ 0.168011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.169011] ************************************************************* [ 0.170603] NET: Registered protocol family 16 [ 0.171445] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.172065] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.173038] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.174425] cpuidle: using governor menu [ 0.176731] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.179631] PCI: Using configuration type 1 for base access [ 0.182123] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.192047] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.193105] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.195036] cryptd: max_cpu_qlen set to 1000 [ 0.196251] ACPI: Added _OSI(Module Device) [ 0.197013] ACPI: Added _OSI(Processor Device) [ 0.198010] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.198986] ACPI: Added _OSI(Processor Aggregator Device) [ 0.202043] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.204598] ACPI: Interpreter enabled [ 0.205058] ACPI: PM: (supports S0 S3 S4 S5) [ 0.206013] ACPI: Using IOAPIC for interrupt routing [ 0.207127] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.208380] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.217819] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.218043] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.219023] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.220121] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.224302] acpiphp: Slot [2] registered [ 0.225097] acpiphp: Slot [5] registered [ 0.227111] acpiphp: Slot [6] registered [ 0.228085] acpiphp: Slot [7] registered [ 0.230112] acpiphp: Slot [8] registered [ 0.231100] acpiphp: Slot [9] registered [ 0.232101] acpiphp: Slot [10] registered [ 0.233133] acpiphp: Slot [3] registered [ 0.234125] acpiphp: Slot [4] registered [ 0.235094] acpiphp: Slot [11] registered [ 0.237088] acpiphp: Slot [12] registered [ 0.238039] acpiphp: Slot [13] registered [ 0.238960] acpiphp: Slot [14] registered [ 0.239210] acpiphp: Slot [15] registered [ 0.240073] acpiphp: Slot [16] registered [ 0.241114] acpiphp: Slot [17] registered [ 0.242166] acpiphp: Slot [18] registered [ 0.243067] acpiphp: Slot [19] registered [ 0.245073] acpiphp: Slot [20] registered [ 0.246000] acpiphp: Slot [21] registered [ 0.247070] acpiphp: Slot [22] registered [ 0.248110] acpiphp: Slot [23] registered [ 0.250561] acpiphp: Slot [24] registered [ 0.251129] acpiphp: Slot [25] registered [ 0.253104] acpiphp: Slot [26] registered [ 0.254097] acpiphp: Slot [27] registered [ 0.256101] acpiphp: Slot [28] registered [ 0.257107] acpiphp: Slot [29] registered [ 0.259106] acpiphp: Slot [30] registered [ 0.260108] acpiphp: Slot [31] registered [ 0.262089] PCI host bridge to bus 0000:00 [ 0.263020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.266025] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.268022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.272030] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.274024] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.277032] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.279175] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.281012] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.284284] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.294727] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.299913] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.302020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.304017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.307021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.310052] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.313112] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.315041] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.317859] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.324018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.339022] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.345013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.349517] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.357020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.363017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.382019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.391000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.399022] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.408023] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.426026] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.439455] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.456027] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.461015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.473022] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.483433] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.492019] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.497019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.513021] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.522691] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.527013] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.532018] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.548017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.555570] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.565021] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.572019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.594017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.603271] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.604297] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.606245] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.607237] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.608164] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.611183] iommu: Default domain type: Passthrough [ 0.613442] SCSI subsystem initialized [ 0.615150] ACPI: bus type USB registered [ 0.616120] usbcore: registered new interface driver usbfs [ 0.618083] usbcore: registered new interface driver hub [ 0.619061] usbcore: registered new device driver usb [ 0.620205] pps_core: LinuxPPS API ver. 1 registered [ 0.621009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.623061] PTP clock support registered [ 0.625031] EDAC MC: Ver: 3.0.0 [ 0.627115] PCI: Using ACPI for IRQ routing [ 0.628593] NetLabel: Initializing [ 0.630008] NetLabel: domain hash size = 128 [ 0.631009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.632129] NetLabel: unlabeled traffic allowed by default [ 0.634122] vgaarb: loaded [ 0.635244] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.636009] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.642299] clocksource: Switched to clocksource kvm-clock [ 0.731846] VFS: Disk quotas dquot_6.6.0 [ 0.733583] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.736083] *** VALIDATE ramfs *** [ 0.737108] *** VALIDATE hugetlbfs *** [ 0.738597] pnp: PnP ACPI init [ 0.740588] pnp: PnP ACPI: found 6 devices [ 0.764043] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.765789] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.766983] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.768726] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.770282] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.772671] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.775313] NET: Registered protocol family 2 [ 0.777412] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.781556] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.783891] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.787660] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.790221] TCP: Hash tables configured (established 65536 bind 65536) [ 0.792493] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.795555] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.798184] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.800554] NET: Registered protocol family 1 [ 0.802800] RPC: Registered named UNIX socket transport module. [ 0.804033] RPC: Registered udp transport module. [ 0.805790] RPC: Registered tcp transport module. [ 0.807499] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.809602] NET: Registered protocol family 44 [ 0.810742] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.811991] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.813550] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.814985] PCI: CLS 0 bytes, default 64 [ 0.816567] Unpacking initramfs... [ 2.203662] debug: unmapping init [mem 0xffff8ff43cc54000-0xffff8ff43ffbffff] [ 2.207415] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.209664] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.211831] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.704689] Initialise system trusted keyrings [ 2.705992] Key type blacklist registered [ 2.707822] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.717233] zbud: loaded [ 2.720440] *** VALIDATE nfs *** [ 2.721794] *** VALIDATE nfs4 *** [ 2.723578] pstore: using deflate compression [ 2.727578] Platform Keyring initialized [ 2.810570] NET: Registered protocol family 38 [ 2.812398] Key type asymmetric registered [ 2.813944] Asymmetric key parser 'x509' registered [ 2.816120] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.820204] io scheduler mq-deadline registered [ 2.821649] io scheduler kyber registered [ 2.823205] io scheduler bfq registered [ 2.824630] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.826741] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.828673] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.830747] ACPI: Power Button [PWRF] [ 2.834642] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.840386] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.854278] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.863967] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.880760] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.914954] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.949181] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.952692] Non-volatile memory driver v1.3 [ 2.953859] Linux agpgart interface v0.103 [ 2.988098] virtio_blk virtio1: [vda] 136584 512-byte logical blocks (69.9 MB/66.7 MiB) [ 2.991470] vda: detected capacity change from 0 to 69931008 [ 3.007912] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.010221] vdb: detected capacity change from 0 to 1073741824 [ 3.028121] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.030269] vdc: detected capacity change from 0 to 2621440000 [ 3.043391] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.045547] vdd: detected capacity change from 0 to 2621440000 [ 3.058546] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.060402] vde: detected capacity change from 0 to 4294967296 [ 3.073370] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.074962] vdf: detected capacity change from 0 to 4294967296 [ 3.079528] libphy: Fixed MDIO Bus: probed [ 3.086841] usbcore: registered new interface driver usbserial_generic [ 3.089543] usbserial: USB Serial support registered for generic [ 3.091180] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.094547] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.095965] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.098261] mousedev: PS/2 mouse device common for all mice [ 3.100559] rtc_cmos 00:05: RTC can wake from S4 [ 3.102963] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.106299] rtc_cmos 00:05: registered as rtc0 [ 3.109891] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.110337] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.113459] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.115273] intel_pstate: CPU model not supported [ 3.120721] hid: raw HID events driver (C) Jiri Kosina [ 3.124117] usbcore: registered new interface driver usbhid [ 3.125277] usbhid: USB HID core driver [ 3.126168] drop_monitor: Initializing network drop monitor service [ 3.127461] Initializing XFRM netlink socket [ 3.128828] NET: Registered protocol family 10 [ 3.130683] Segment Routing with IPv6 [ 3.131512] NET: Registered protocol family 17 [ 3.132630] mpls_gso: MPLS GSO support [ 3.138157] RAS: Correctable Errors collector initialized. [ 3.139411] AVX version of gcm_enc/dec engaged. [ 3.140407] AES CTR mode by8 optimization enabled [ 3.215271] sched_clock: Marking stable (3215251154, 0)->(4137204678, -921953524) [ 3.217751] registered taskstats version 1 [ 3.219100] Loading compiled-in X.509 certificates [ 3.220132] zswap: loaded using pool lzo/zbud [ 3.240077] Key type big_key registered [ 3.250759] Key type encrypted registered [ 3.252351] ima: No TPM chip found, activating TPM-bypass! [ 3.254373] ima: Allocated hash algorithm: sha1 [ 3.256086] ima: No architecture policies found [ 3.257566] evm: Initialising EVM extended attributes: [ 3.259218] evm: security.selinux [ 3.260424] evm: security.ima [ 3.261296] evm: security.capability [ 3.262440] evm: HMAC attrs: 0x1 [ 3.264580] rtc_cmos 00:05: setting system clock to 2026-06-12 04:01:23 UTC (1781236883) [ 3.270902] debug: unmapping init [mem 0xffffffff8a003000-0xffffffff8a1fffff] [ 3.273160] debug: unmapping init [mem 0xffffffff88d82000-0xffffffff89058fff] [ 3.282115] Write protecting the kernel read-only data: 28672k [ 3.286427] debug: unmapping init [mem 0xffffffff87403000-0xffffffff875fffff] [ 3.288964] debug: unmapping init [mem 0xffffffff87d14000-0xffffffff87dfffff] [ 3.321571] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.329500] systemd[1]: Detected virtualization kvm. [ 3.331065] systemd[1]: Detected architecture x86-64. [ 3.332648] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.358040] systemd[1]: No hostname configured. [ 3.359559] systemd[1]: Set hostname to . [ 3.361458] random: systemd: uninitialized urandom read (16 bytes read) [ 3.363416] systemd[1]: Initializing machine ID from random generator. [ 3.405567] random: ln: uninitialized urandom read (6 bytes read) [ 3.483531] random: systemd: uninitialized urandom read (16 bytes read) [ 3.485823] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.489666] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.493102] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.016087] device-mapper: uevent: version 1.0.3 [ 4.018290] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.555870] random: fast init done Starting dracut initqueue hook... [ 4.582852] virtio_net virtio0 ens2: renamed from eth0 [ 4.645465] scsi host0: ata_piix [ 4.669293] scsi host1: ata_piix [ 4.671130] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.673863] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.895984] dracut-initqueue[586]: RTNETLINK answers: File exists [ 9.766851] random: crng init done [ 9.768148] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.013273] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.011933] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.247040] SELinux: Disabled at runtime. [ 11.301210] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.306452] systemd[1]: Detected virtualization kvm. [ 11.307448] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.697994] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.700369] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.704430] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.708378] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.711218] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.717493] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.720553] systemd[1]: Listening on initctl Compatibility Named Pipe. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-getty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice User and Session Slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-serial\x2dgetty.slice. Mounting POSIX Message Queue File System... Mounting Huge Pages File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Mounting Kernel Debug File System... Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Starting Apply Kernel Variables... Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target Slices. [ OK ] Started Journal Service. [ 11.870616] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.101711] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.336975] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.344529] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.471567] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.485387] EDAC sbridge: Ver: 1.1.2 [ 14.023803] Key type dns_resolver registered [ 14.317753] NFS: Registering the id_resolver key type [ 14.319710] Key type id_resolver registered [ 14.321034] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg116-server login: [ 39.347734] libcfs: loading out-of-tree module taints kernel. [ 39.368749] Key type ._llcrypt registered [ 39.370239] Key type .llcrypt registered [ 39.421753] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 48.389744] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 48.964467] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 48.971660] alg: No test for adler32 (adler32-zlib) [ 50.000048] Lustre: Lustre: Build Version: 2.17.53_79_gff07dcf [ 50.327615] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 51.951180] Key type lgssc registered [ 53.697426] Lustre: Echo OBD driver; http://www.lustre.org/ [ 79.063689] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 127.674372] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 142.183257] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 142.236888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 143.694213] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 143.748797] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 143.919819] Lustre: lustre-MDT0000: new disk, initializing [ 144.122572] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 144.164479] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 147.348131] hrtimer: interrupt took 3121942 ns [ 149.568144] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 164.207807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 164.325971] Lustre: 6480:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 164.375493] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 164.382930] Lustre: Skipped 1 previous similar message [ 164.471192] Lustre: lustre-MDT0001: new disk, initializing [ 164.562152] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 164.604197] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 164.619707] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 169.550531] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 174.915899] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 186.747167] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 187.131276] Lustre: lustre-OST0000: new disk, initializing [ 187.136780] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 187.145435] Lustre: 8383:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 187.287462] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 189.612929] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 189.630619] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 189.758423] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 194.076575] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 209.412820] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 209.740415] Lustre: lustre-OST0001: new disk, initializing [ 209.747325] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 209.759332] Lustre: 9440:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 209.845580] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 217.003303] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 219.205146] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 219.213680] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 219.275903] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 229.478591] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 237.545317] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 244.954463] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing check_logdir /tmp/testlogs/ [ 250.469229] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing yml_node [ 255.245242] Lustre: DEBUG MARKER: Client: 2.17.53.79 [ 258.353983] Lustre: DEBUG MARKER: MDS: 2.17.53.79 [ 261.254919] Lustre: DEBUG MARKER: OSS: 2.17.53.79 [ 263.171366] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Jun 12 00:05:42 EDT 2026 [ 283.941390] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 286.030896] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 306.665162] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 306.666776] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 306.697067] Lustre: Skipped 3 previous similar messages [ 306.715607] Lustre: Skipped 3 previous similar messages [ 309.615264] Lustre: server umount lustre-MDT0000 complete [ 315.195961] LustreError: 7411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781237195 with bad export cookie 8916247676358223009 [ 315.198199] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 315.220452] LustreError: 7411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 316.901139] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 316.902150] LustreError: 6486:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 316.902644] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 316.911994] Lustre: Skipped 1 previous similar message [ 316.964230] LustreError: 6486:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 320.931391] LustreError: 8503:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 320.932714] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 320.943914] LustreError: 8503:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 320.960228] Lustre: Skipped 2 previous similar messages [ 321.923627] Lustre: server umount lustre-MDT0001 complete [ 327.255257] Lustre: server umount lustre-OST0000 complete [ 332.721525] Lustre: server umount lustre-OST0001 complete [ 340.048097] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 352.228341] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 392.965051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 393.290443] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 393.331913] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 393.421454] Lustre: lustre-MDT0000: new disk, initializing [ 393.525125] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 393.544901] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 398.089769] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 410.195560] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 410.404971] Lustre: 14961:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 410.466777] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 410.484055] Lustre: Skipped 1 previous similar message [ 410.630281] Lustre: lustre-MDT0001: new disk, initializing [ 410.796596] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 410.828640] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 410.835406] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 415.618496] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 424.068951] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 429.214086] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 436.434675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 436.753875] Lustre: lustre-OST0000: new disk, initializing [ 436.759610] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 436.766957] Lustre: 16703:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 436.887488] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 438.204715] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 438.209931] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 438.305389] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 443.944449] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 455.115970] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 462.881426] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 463.207147] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 469.609964] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 469.816283] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 474.082643] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 474.093050] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 474.105982] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 474.111990] Lustre: Skipped 1 previous similar message [ 477.820863] Lustre: server umount lustre-OST0000 complete [ 482.274735] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 482.290733] LustreError: Skipped 1 previous similar message [ 482.302522] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 482.313700] Lustre: Skipped 1 previous similar message [ 482.318216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 488.246684] Lustre: server umount lustre-MDT0000 complete [ 489.441554] LustreError: 14967:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 489.463438] LustreError: 14967:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 492.407991] LustreError: 14952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781237372 with bad export cookie 8916247676358224535 [ 492.419188] LustreError: 14952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 492.422074] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 492.913635] Lustre: server umount lustre-MDT0001 complete [ 499.608604] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 00:09:39 (1781237379) [ 519.540754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 519.981096] LustreError: 19431:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 520.044872] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 524.409694] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 525.291381] LustreError: 19432:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 525.304106] LustreError: 19432:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 533.463828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 537.945798] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 544.033261] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 547.722271] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 550.022390] Lustre: Failing over lustre-MDT0000 [ 550.033632] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 550.038991] Lustre: Skipped 1 previous similar message [ 550.041566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 550.044601] Lustre: Skipped 3 previous similar messages [ 550.387525] Lustre: server umount lustre-MDT0000 complete [ 555.489414] LustreError: 20154:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 555.508497] LustreError: 20154:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 570.559593] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 570.705975] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 575.248157] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 575.974306] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 576.003409] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 576.039194] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 583.616297] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 591.331755] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 591.334252] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 591.343887] Lustre: Skipped 2 previous similar messages [ 591.345211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 594.186051] Lustre: server umount lustre-MDT0000 complete [ 596.452525] LustreError: 19426:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 596.477293] LustreError: 19426:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 597.853622] LustreError: 19413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781237478 with bad export cookie 8916247676358225956 [ 597.864331] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 597.868716] LustreError: 19413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 598.248733] Lustre: server umount lustre-MDT0001 complete [ 607.033116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 607.751705] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 607.768962] Lustre: Skipped 2 previous similar messages [ 612.947876] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 621.779553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 626.959353] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 633.515734] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 637.696899] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 640.225215] Lustre: Failing over lustre-MDT0000 [ 640.486988] Lustre: server umount lustre-MDT0000 complete [ 642.528210] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 642.529527] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 642.548394] LustreError: 22429:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 642.551898] Lustre: Skipped 1 previous similar message [ 642.589236] LustreError: 22429:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 657.887180] Lustre: 3627:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781237522/real 1781237522] req@ffff8ff382fd4b40 x1867762295304320/t0(0) o400->MGC192.168.201.116@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1781237538 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 657.921430] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 660.513953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 668.403954] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 673.762266] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 673.773241] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 673.778193] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 673.778203] Lustre: Skipped 1 previous similar message [ 673.813757] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 683.218639] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 687.546529] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 687.563367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 687.569764] Lustre: Skipped 1 previous similar message [ 693.867839] Lustre: server umount lustre-MDT0000 complete [ 697.965493] LustreError: 22411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781237578 with bad export cookie 8916247676358226894 [ 697.986162] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 697.997182] LustreError: 22411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 698.374994] Lustre: server umount lustre-MDT0001 complete [ 708.150489] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 00:13:07 (1781237587) [ 725.827353] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 735.714967] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 781.110281] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 793.743055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 794.139540] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 794.175492] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 794.304909] Lustre: lustre-MDT0000: new disk, initializing [ 794.510446] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 794.513260] Lustre: Skipped 2 previous similar messages [ 794.572888] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 800.519489] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 813.133550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 813.358704] Lustre: 28420:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 813.760895] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 819.123763] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 828.647129] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 833.564809] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 842.123660] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 842.529797] Lustre: 30158:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 844.030474] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 844.110640] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 844.252556] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 844.295466] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 851.166634] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 860.063431] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 867.170936] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 890.344710] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 890.348891] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 890.361037] LustreError: Skipped 1 previous similar message [ 890.363071] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 890.363078] Lustre: Skipped 4 previous similar messages [ 890.392972] Lustre: Skipped 2 previous similar messages [ 895.842361] Lustre: server umount lustre-OST0000 complete [ 900.583245] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 906.967347] Lustre: server umount lustre-MDT0000 complete [ 910.821828] LustreError: 28430:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 910.849887] LustreError: 28430:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 13 previous similar messages [ 911.715778] LustreError: 28412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781237791 with bad export cookie 8916247676358227300 [ 911.734963] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 912.254049] Lustre: server umount lustre-MDT0001 complete [ 925.966045] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 929.090451] Key type lgssc unregistered [ 929.475970] LNet: 32012:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 929.483650] LNetError: 32012:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 929.507071] LNet: Removed LNI 192.168.201.116@tcp [ 930.761153] Key type .llcrypt unregistered [ 930.762636] Key type ._llcrypt unregistered [ 959.677418] Key type ._llcrypt registered [ 959.679841] Key type .llcrypt registered [ 959.833924] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 961.193597] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 961.381261] alg: No test for adler32 (adler32-zlib) [ 962.695936] Lustre: Lustre: Build Version: 2.17.53_79_gff07dcf [ 963.055584] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 964.836808] Key type lgssc registered [ 965.999396] Lustre: Echo OBD driver; http://www.lustre.org/ [ 975.161684] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 975.184364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 976.798382] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 982.190861] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 987.114197] LustreError: 33410:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 987.140404] LustreError: 33410:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 991.290870] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 991.763587] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 996.769570] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1002.707519] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1007.215698] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1014.605514] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1015.123469] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1016.144955] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1022.396246] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1022.513504] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1029.311087] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1045.488622] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 00:18:44 (1781237924) [ 1047.455443] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1049.306262] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1056.976461] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 00:18:56 (1781237936) [ 1066.194793] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 00:19:05 (1781237945) [ 1067.402513] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1076.753719] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1076.756608] Lustre: Skipped 2 previous similar messages [ 1078.815610] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 00:19:18 (1781237958) [ 1082.292507] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1082.299020] Lustre: Skipped 3 previous similar messages [ 1101.853242] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1101.860476] Lustre: Skipped 6 previous similar messages [ 1104.831393] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 00:19:44 (1781237984) [ 1108.183978] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1112.126740] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1122.353813] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 00:20:01 (1781238001) [ 1124.425326] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1124.432739] Lustre: Skipped 3 previous similar messages [ 1140.514752] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 00:20:19 (1781238019) [ 1159.649474] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1159.655843] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1159.669848] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1160.675667] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1160.690980] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1160.698119] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1160.731735] Lustre: Skipped 1 previous similar message [ 1163.089535] Lustre: server umount lustre-OST0000 complete [ 1170.914119] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1170.917071] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1170.919074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1170.938391] Lustre: Skipped 1 previous similar message [ 1170.955866] Lustre: Skipped 1 previous similar message [ 1174.630728] Lustre: server umount lustre-MDT0000 complete [ 1176.036523] LustreError: 33406:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1176.063123] LustreError: 33406:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1179.037615] LustreError: 33392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781238059 with bad export cookie 12498251100974032568 [ 1179.041329] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1179.049420] LustreError: 33392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1179.438413] Lustre: server umount lustre-MDT0001 complete [ 1191.208545] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 1194.745981] Key type lgssc unregistered [ 1195.199604] LNet: 40914:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1195.207205] LNetError: 40914:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1195.232474] LNet: Removed LNI 192.168.201.116@tcp [ 1196.415858] Key type .llcrypt unregistered [ 1196.418845] Key type ._llcrypt unregistered [ 1218.276292] Key type ._llcrypt registered [ 1218.277950] Key type .llcrypt registered [ 1218.468071] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 1235.035283] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 1236.633976] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1236.678484] alg: No test for adler32 (adler32-zlib) [ 1237.950790] Lustre: Lustre: Build Version: 2.17.53_79_gff07dcf [ 1238.227120] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 1239.960871] Key type lgssc registered [ 1241.296636] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1283.004575] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1283.022830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1284.380222] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1284.453131] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1284.709621] Lustre: lustre-MDT0000: new disk, initializing [ 1285.052788] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1285.085867] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1289.872636] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1303.342340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1303.432061] Lustre: 44037:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1303.505446] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1303.514818] Lustre: Skipped 1 previous similar message [ 1303.640749] Lustre: lustre-MDT0001: new disk, initializing [ 1303.754879] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1303.810346] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1303.818099] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1308.904453] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1318.267664] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1321.688390] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1329.791457] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1330.367115] Lustre: lustre-OST0000: new disk, initializing [ 1330.375119] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1330.389552] Lustre: 45779:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1330.558925] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1338.982398] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1339.007380] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1339.234622] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1339.460386] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1348.555902] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1353.058853] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1353.324143] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1357.823881] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1358.019778] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1359.841155] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1359.841552] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1359.847081] LustreError: Skipped 1 previous similar message [ 1359.859313] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1364.966829] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1364.970732] Lustre: Skipped 1 previous similar message [ 1366.039279] Lustre: server umount lustre-OST0000 complete [ 1375.203508] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1375.216430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1375.217137] Lustre: Skipped 2 previous similar messages [ 1375.233437] Lustre: Skipped 2 previous similar messages [ 1377.173040] Lustre: server umount lustre-MDT0000 complete [ 1380.887462] LustreError: 44029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781238261 with bad export cookie 18364740259397942307 [ 1380.892126] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1380.895353] LustreError: 44029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1381.241186] Lustre: server umount lustre-MDT0001 complete [ 1392.719637] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 00:24:31 (1781238271) [ 1401.574679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1402.214145] LustreError: 48091:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1402.245248] LustreError: 48091:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1402.335556] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1407.384072] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1407.464622] LustreError: 48092:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1412.603403] LustreError: 48091:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1417.327435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1417.618758] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1422.340310] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1428.495372] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1433.332474] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1440.501233] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1441.083157] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1449.500105] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1456.043331] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1460.792917] LustreError: 50281:0:(llog.c:444:llog_init_handle()) llog type is not specified! [ 1461.937130] Lustre: Modifying parameter general.timeout=1 in log params [ 1462.989491] Lustre: Modifying parameter general.timeout=2 in log params [ 1465.434847] Lustre: Modifying parameter general.timeout=5 in log params [ 1465.437762] Lustre: Skipped 2 previous similar messages [ 1469.796616] Lustre: Modifying parameter general.timeout=10 in log params [ 1469.801455] Lustre: Skipped 4 previous similar messages [ 1478.432075] Lustre: Modifying parameter general.timeout=19 in log params [ 1478.436350] Lustre: Skipped 8 previous similar messages [ 1495.652096] Lustre: Modifying parameter general.timeout=37 in log params [ 1495.657402] Lustre: Skipped 17 previous similar messages [ 1528.185234] Lustre: Modifying parameter general.timeout=68 in log params [ 1528.189319] Lustre: Skipped 30 previous similar messages [ 1599.794837] Lustre: Disabling parameter general.timeout= in log params [ 1599.802583] Lustre: Skipped 61 previous similar messages [ 1603.094621] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 00:28:02 (1781238482) [ 1606.495027] Lustre: 57230:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1608.596543] Lustre: 57328:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1610.926274] Lustre: 57441:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1615.219599] Lustre: 57637:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=50 (mode = 2) failed: rc = -17 [ 1615.228751] Lustre: 57637:0:(mgs_llog.c:1437:mgs_modify_param()) Skipped 1 previous similar message [ 1644.902675] Lustre: 59106:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1644.916829] Lustre: 59106:0:(mgs_llog.c:1437:mgs_modify_param()) Skipped 1 previous similar message [ 1644.966565] LustreError: 59106:0:(mgs_handler.c:1202:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1649.806880] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 00:28:48 (1781238528) [ 1668.066120] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1668.073713] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1668.085522] Lustre: Skipped 2 previous similar messages [ 1668.098038] Lustre: Skipped 1 previous similar message [ 1669.897566] Lustre: server umount lustre-MDT0000 complete [ 1673.193683] LustreError: 48803:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1673.209104] LustreError: 48803:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1673.822877] LustreError: 48070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781238554 with bad export cookie 18364740259397943189 [ 1673.836794] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1674.159931] Lustre: server umount lustre-MDT0001 complete [ 1688.867191] Lustre: server umount lustre-OST0000 complete [ 1717.872432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1718.288490] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1718.594273] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1723.355466] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1733.587943] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1733.687907] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1733.727766] Lustre: 61638:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1739.058811] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1744.916809] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1748.369893] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1754.700771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1754.890772] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1755.200094] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1755.204591] Lustre: Skipped 1 previous similar message [ 1762.084939] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1768.738509] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1770.525536] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1770.537586] Lustre: Skipped 23 previous similar messages [ 1770.596062] Lustre: 63865:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1782.752835] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1782.765421] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1782.785851] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1782.791772] Lustre: Skipped 1 previous similar message [ 1784.406140] Lustre: server umount lustre-OST0000 complete [ 1788.386815] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1788.414702] Lustre: Skipped 2 previous similar messages [ 1793.504350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1793.509633] Lustre: Skipped 5 previous similar messages [ 1794.444456] Lustre: server umount lustre-MDT0000 complete [ 1798.400032] LustreError: 62363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781238678 with bad export cookie 18364740259397946346 [ 1798.404050] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1798.411047] LustreError: 62363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 1798.788917] Lustre: server umount lustre-MDT0001 complete [ 1811.620385] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 1815.343625] Key type lgssc unregistered [ 1815.627831] LNet: 65055:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1815.633445] LNetError: 65055:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1816.679969] LNet: Removed LNI 192.168.201.116@tcp [ 1817.113540] Key type .llcrypt unregistered [ 1817.114954] Key type ._llcrypt unregistered [ 1846.348872] Key type ._llcrypt registered [ 1846.351235] Key type .llcrypt registered [ 1846.495229] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 1847.551676] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1847.595528] alg: No test for adler32 (adler32-zlib) [ 1848.625993] Lustre: Lustre: Build Version: 2.17.53_79_gff07dcf [ 1848.774302] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 1850.423188] Key type lgssc registered [ 1851.964912] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1860.488263] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1860.522892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1862.778664] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1868.548993] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1872.352826] LustreError: 66461:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1872.380650] LustreError: 66461:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1877.473473] LustreError: 66460:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1877.747337] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1878.439148] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1883.968720] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1890.050405] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1894.310433] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1901.612957] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1902.129187] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1908.761140] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1914.812259] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1922.893957] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1933.290113] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1933.297501] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1933.299017] Lustre: Skipped 1 previous similar message [ 1933.320913] Lustre: Skipped 2 previous similar messages [ 1938.341191] Lustre: server umount lustre-MDT0000 complete [ 1938.405345] LustreError: 68082:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1938.423951] LustreError: 68082:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1941.971705] LustreError: 68085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781238822 with bad export cookie 335904141702007954 [ 1941.980485] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1941.986497] LustreError: 68085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1942.320431] Lustre: server umount lustre-MDT0001 complete [ 1957.028912] Lustre: server umount lustre-OST0000 complete [ 1987.952311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1988.183616] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1988.203697] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1988.453031] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1993.642700] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2003.988260] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2004.201966] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2004.231396] Lustre: 70990:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2004.837469] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2010.324446] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2017.654538] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2022.935508] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2031.336277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2031.722020] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2032.135647] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2039.253985] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2045.012863] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2046.580387] Lustre: 73230:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2048.370995] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2055.656666] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2055.673603] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2055.683564] Lustre: Skipped 1 previous similar message [ 2055.691607] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2058.989231] Lustre: server umount lustre-OST0000 complete [ 2065.889269] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2065.902485] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2065.921528] Lustre: Skipped 1 previous similar message [ 2065.927728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2065.937839] Lustre: Skipped 2 previous similar messages [ 2071.522392] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2071.530491] Lustre: Skipped 3 previous similar messages [ 2076.646782] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2076.649873] Lustre: Skipped 1 previous similar message [ 2077.151759] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2077.410832] Lustre: server umount lustre-MDT0000 complete [ 2081.571907] LustreError: 70984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781238961 with bad export cookie 335904141702008801 [ 2081.579796] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2081.589865] LustreError: 70984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2081.904963] Lustre: server umount lustre-MDT0001 complete [ 2092.594888] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 2095.303944] Key type lgssc unregistered [ 2095.549830] LNet: 74426:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2095.558994] LNetError: 74426:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2095.601490] LNet: Removed LNI 192.168.201.116@tcp [ 2096.520387] Key type .llcrypt unregistered [ 2096.522712] Key type ._llcrypt unregistered [ 2121.018310] Key type ._llcrypt registered [ 2121.020154] Key type .llcrypt registered [ 2121.113489] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 2122.376338] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2122.416840] alg: No test for adler32 (adler32-zlib) [ 2123.480651] Lustre: Lustre: Build Version: 2.17.53_79_gff07dcf [ 2123.688654] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 2125.415537] Key type lgssc registered [ 2126.610476] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2134.768738] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2134.777213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2136.076680] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2141.131954] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2146.274719] LustreError: 75823:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2146.295457] LustreError: 75823:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2151.043104] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2151.548884] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2157.019118] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2163.290419] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2167.733584] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2175.527160] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2176.082309] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2183.405951] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2189.941747] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2196.587247] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2197.421317] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2199.263895] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2199.273487] Lustre: Skipped 1 previous similar message [ 2201.311437] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2201.328348] Lustre: Skipped 1 previous similar message [ 2205.949318] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2205.956232] Lustre: Skipped 4 previous similar messages [ 2214.674957] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2214.677021] Lustre: Skipped 9 previous similar messages [ 2231.620592] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2231.627478] Lustre: Skipped 16 previous similar messages [ 2264.192405] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2264.222262] Lustre: Skipped 33 previous similar messages [ 2297.847098] Lustre: 83586:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2300.966834] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 00:39:40 (1781239180) [ 2303.031161] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2304.862615] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 00:39:44 (1781239184) [ 2309.601288] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2309.617182] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2309.631279] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2309.636902] Lustre: Skipped 1 previous similar message [ 2314.721978] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2314.734388] Lustre: Skipped 1 previous similar message [ 2315.205817] Lustre: server umount lustre-OST0000 complete [ 2319.842682] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2319.862034] Lustre: Skipped 1 previous similar message [ 2319.880142] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2319.889650] Lustre: Skipped 1 previous similar message [ 2324.820914] Lustre: server umount lustre-MDT0000 complete [ 2324.961961] LustreError: 75823:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2324.961961] LustreError: 75818:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2328.221594] LustreError: 83589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781239208 with bad export cookie 15284326043116575544 [ 2328.231118] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2328.232939] LustreError: 83589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2328.672614] Lustre: server umount lustre-MDT0001 complete [ 2339.613315] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 2342.517210] Key type lgssc unregistered [ 2342.793313] LNet: 84820:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2342.807800] LNetError: 84820:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2342.827684] LNet: Removed LNI 192.168.201.116@tcp [ 2343.613396] Key type .llcrypt unregistered [ 2343.615410] Key type ._llcrypt unregistered [ 2348.177914] Key type ._llcrypt registered [ 2348.179375] Key type .llcrypt registered [ 2348.321775] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2353.184519] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules [ 2354.521993] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2354.574104] alg: No test for adler32 (adler32-zlib) [ 2355.669933] Lustre: Lustre: Build Version: 2.17.53_79_gff07dcf [ 2355.906879] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 2357.639171] Key type lgssc registered [ 2358.902685] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2359.963643] LustreError: 85996:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 2376.737279] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 2381.348245] LustreError: 85996:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2386.073705] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2386.091424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2387.635423] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2391.649676] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2397.166382] LustreError: 86996:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2397.181259] LustreError: 86996:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2402.108854] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 00:41:21 (1781239281) [ 2402.281708] LustreError: 86995:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2406.580561] LustreError: 87027:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2406.585153] LustreError: 87027:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2406.589564] LustreError: 87027:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 2407.324089] Lustre: server umount lustre-MDT0000 complete [ 2422.630493] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 2425.839828] Key type lgssc unregistered [ 2426.099405] LNet: 88487:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2426.108486] LNetError: 88487:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2426.119459] LNet: Removed LNI 192.168.201.116@tcp [ 2426.916794] Key type .llcrypt unregistered [ 2426.926339] Key type ._llcrypt unregistered [ 2445.728985] Key type ._llcrypt registered [ 2445.731469] Key type .llcrypt registered [ 2445.910207] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 2447.469282] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2447.553037] alg: No test for adler32 (adler32-zlib) [ 2448.723783] Lustre: Lustre: Build Version: 2.17.53_79_gff07dcf [ 2448.981267] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 2450.657203] Key type lgssc registered [ 2451.801343] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2460.989717] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2461.036468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2462.653232] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2468.103875] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2471.909488] LustreError: 89688:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2471.930920] LustreError: 89688:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2477.032240] LustreError: 89689:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2478.326543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2479.179624] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2484.110687] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2489.820389] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2493.378196] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2500.367410] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2500.872089] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2506.913790] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2512.808769] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2564.716277] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 00:44:04 (1781239444) [ 2566.693733] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2568.608477] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 00:44:07 (1781239447) [ 2573.793606] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2573.800770] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2573.824348] Lustre: Skipped 1 previous similar message [ 2578.913185] LustreError: 89688:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2579.089958] Lustre: server umount lustre-MDT0000 complete [ 2582.635302] LustreError: 89668:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781239462 with bad export cookie 8777476071713509729 [ 2582.643416] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2582.991290] Lustre: server umount lustre-MDT0001 complete [ 2596.232706] Lustre: server umount lustre-OST0000 complete [ 2607.870784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2608.306639] LustreError: 93640:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2608.329579] LustreError: 93640:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2608.491412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2613.079175] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2613.728781] LustreError: 93641:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2622.455283] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2623.148912] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2628.829883] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2636.358701] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2641.840078] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2658.240289] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2658.497416] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2658.505592] LustreError: 93630:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2658.543323] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2658.564272] LustreError: 95499:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -98 [ 2658.571101] LustreError: 95499:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 2658.576312] LustreError: 95499:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2658.702487] Lustre: server umount lustre-OST0000 complete [ 2658.704303] LustreError: 95499:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2670.365321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2670.631481] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2670.642679] LustreError: 93627:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2670.674855] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2670.689602] LustreError: 95965:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -98 [ 2670.700888] LustreError: 95965:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 2670.710181] LustreError: 95965:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2670.831984] Lustre: server umount lustre-OST0000 complete [ 2670.839453] LustreError: 95965:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2682.473756] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2682.619414] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2682.623575] Lustre: Found index 0 for lustre-OST0000, updating log [ 2682.627831] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2682.696377] Lustre: lustre-OST0000: new disk, initializing [ 2682.706170] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2682.715900] Lustre: 96501:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2682.972088] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2690.117039] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2694.184337] Lustre: Failing over lustre-OST0000 [ 2694.341877] Lustre: server umount lustre-OST0000 complete [ 2698.368878] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2698.370746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2698.382857] Lustre: Skipped 2 previous similar messages [ 2698.392756] Lustre: Skipped 3 previous similar messages [ 2703.842392] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2704.921152] Lustre: server umount lustre-MDT0000 complete [ 2708.962417] LustreError: 93636:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2708.979286] LustreError: 93636:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2709.558828] LustreError: 95519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781239589 with bad export cookie 8777476071713512473 [ 2709.562639] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2709.571826] LustreError: 95519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 2719.359234] Lustre: server umount lustre-MDT0001 complete [ 2729.494753] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 00:46:48 (1781239608) [ 2749.127908] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 2759.920559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2760.464218] LustreError: 98814:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2760.697035] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2765.451282] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2776.215966] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2784.334725] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2788.577871] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2799.853648] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2800.596422] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2800.614846] Lustre: Skipped 1 previous similar message [ 2800.646212] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2802.608574] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2802.657496] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2802.667728] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2808.575516] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2819.891711] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2820.106104] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2820.339359] Lustre: lustre-OST0001: new disk, initializing [ 2820.343674] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2820.367984] Lustre: 101082:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2829.344067] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2829.350081] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2829.456953] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2829.696790] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2837.041858] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2844.694632] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2852.579243] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.16@tcp (stopping) [ 2852.587917] Lustre: Skipped 1 previous similar message [ 2854.368317] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2854.375158] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2856.438638] Lustre: server umount lustre-MDT0001 complete [ 2857.660898] LustreError: 98809:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2857.687831] LustreError: 98809:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2867.679689] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2867.812309] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2867.816174] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2867.883949] Lustre: 98804:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2868.775615] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2868.791366] Lustre: Skipped 1 previous similar message [ 2869.537522] LustreError: 102609:0:(obd_mount.c:225:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2869.553025] LustreError: 102609:0:(tgt_mount.c:668:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2869.569858] LustreError: 98806:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2869.700764] LustreError: 102620:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.116@tcp: cfg command failed: rc = -17 [ 2869.724858] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2869.724858] [ 2873.841260] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2874.341671] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2885.392930] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 00:49:24 (1781239764) [ 2906.153313] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 2925.638491] Lustre: 104651:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2949.138326] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2953.996388] Lustre: 105785:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3047.403571] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3047.430559] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3047.459232] Lustre: Skipped 2 previous similar messages [ 3047.465215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3047.469167] Lustre: Skipped 3 previous similar messages [ 3048.628810] Lustre: server umount lustre-MDT0000 complete [ 3052.877816] LustreError: 102559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781239933 with bad export cookie 8777476071713513474 [ 3052.882391] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3052.889684] LustreError: 102559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3053.023760] LustreError: 98811:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3053.023997] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3053.050935] LustreError: 98811:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 3053.088097] Lustre: Skipped 4 previous similar messages [ 3053.550533] Lustre: server umount lustre-MDT0001 complete [ 3061.053375] Lustre: server umount lustre-OST0000 complete [ 3065.400438] Lustre: server umount lustre-OST0001 complete [ 3071.264379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3081.424962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3091.110984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3102.629243] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3123.253832] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 3134.208880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3134.243233] Lustre: lustre-MDT0000: reset Object Index mappings [ 3135.053382] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3139.735095] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3150.276934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3150.301348] Lustre: lustre-MDT0001: reset Object Index mappings [ 3155.490952] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3158.624793] Lustre: 110831:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3166.298551] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3174.218989] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3175.235490] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:227 to 0x280000400:257) [ 3180.335989] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:354 to 0x280000401:386) [ 3185.529967] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3185.644918] LustreError: 111757:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3185.670614] LustreError: 111757:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 3191.824369] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 3191.863213] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 3194.411639] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3204.188601] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3207.623816] Lustre: 112678:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3259.228241] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 00:55:38 (1781240138) [ 3289.067559] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3289.077616] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3289.094544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3289.094552] Lustre: Skipped 5 previous similar messages [ 3289.127261] Lustre: Skipped 3 previous similar messages [ 3291.510551] Lustre: server umount lustre-MDT0000 complete [ 3295.247975] LustreError: 109699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781240175 with bad export cookie 8777476071713560339 [ 3295.254722] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3295.266936] LustreError: 109699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3301.956488] Lustre: server umount lustre-MDT0001 complete [ 3313.932118] Lustre: server umount lustre-OST0000 complete [ 3324.575660] Lustre: server umount lustre-OST0001 complete [ 3331.979514] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 3343.935210] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 3385.451559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3385.634433] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3385.649564] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3385.710866] Lustre: lustre-MDT0000: new disk, initializing [ 3385.805859] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3389.855927] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3398.030114] Lustre: server umount lustre-MDT0000 complete [ 3410.476877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3411.239198] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3411.293417] Lustre: 116896:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3411.319902] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3411.769522] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3411.780270] Lustre: Skipped 4 previous similar messages [ 3418.528586] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3423.429916] Lustre: server umount lustre-MDT0000 complete [ 3433.710682] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 00:58:33 (1781240313) [ 3435.770926] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3437.463350] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 00:58:37 (1781240317) [ 3453.598427] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 3462.343726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3467.031915] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3475.151758] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3475.308266] Lustre: 118914:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3475.381698] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3475.386561] Lustre: Skipped 1 previous similar message [ 3475.498571] Lustre: lustre-MDT0001: new disk, initializing [ 3475.636891] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3475.654766] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3480.321454] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3485.733611] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3493.246118] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3493.714687] Lustre: lustre-OST0000: new disk, initializing [ 3493.736328] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3493.749138] Lustre: 120457:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3495.695204] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3495.710441] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3495.838543] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3499.954372] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3511.844230] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3512.046709] Lustre: lustre-OST0001: new disk, initializing [ 3512.053704] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3512.087790] Lustre: 121310:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3513.538791] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3513.552629] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3513.643639] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3519.528850] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3530.606929] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3534.645960] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3919.744931] Lustre: 127723:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3919.750412] Lustre: 127723:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3919.755963] Lustre: 127723:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3919.762442] Lustre: 127723:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3919.770192] Lustre: 127723:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3919.783866] Lustre: 127723:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4208.947417] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 01:11:27 (1781241087) [ 4225.508335] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4225.527059] Lustre: Skipped 2 previous similar messages [ 4225.545710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4225.558888] Lustre: Skipped 7 previous similar messages [ 4226.720677] Lustre: server umount lustre-MDT0000 complete [ 4230.564136] LustreError: 127736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781241110 with bad export cookie 8777476071713628498 [ 4230.570307] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4230.577256] LustreError: 127736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4230.624096] LustreError: 120151:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4230.626069] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4230.631911] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4230.631920] Lustre: Skipped 3 previous similar messages [ 4230.645417] LustreError: 120151:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 4230.674504] Lustre: Skipped 4 previous similar messages [ 4235.752179] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4235.760763] Lustre: Skipped 2 previous similar messages [ 4237.574651] Lustre: server umount lustre-MDT0001 complete [ 4249.304030] Lustre: server umount lustre-OST0000 complete [ 4258.704685] Lustre: server umount lustre-OST0001 complete [ 4266.720573] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 4279.668795] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 4320.312318] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4320.624975] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4320.668568] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4320.768425] Lustre: lustre-MDT0000: new disk, initializing [ 4320.909142] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4320.915687] Lustre: Skipped 4 previous similar messages [ 4320.943984] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4326.496669] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4337.967458] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4338.156202] Lustre: 130749:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4338.235794] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4338.244358] Lustre: Skipped 1 previous similar message [ 4338.409936] Lustre: lustre-MDT0001: new disk, initializing [ 4338.632419] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4338.646353] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4343.334184] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4351.247776] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4355.038244] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4362.070359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4362.561190] Lustre: lustre-OST0000: new disk, initializing [ 4362.564908] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4362.570816] Lustre: 132490:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4364.390147] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4364.405063] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4364.657150] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4369.634531] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4378.391304] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4390.608734] Lustre: lustre-MDD0000: changelog on [ 4393.714760] Lustre: lustre-MDD0001: changelog on [ 6369.805395] Lustre: lustre-MDD0001: changelog off [ 6371.734485] Lustre: 130756:0:(llog_cat.c:977:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6372.703365] Lustre: lustre-MDD0000: changelog off [ 6382.051157] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6382.054147] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6382.058561] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6382.069212] LustreError: Skipped 1 previous similar message [ 6382.097174] Lustre: Skipped 1 previous similar message [ 6387.180109] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6387.196869] Lustre: Skipped 2 previous similar messages [ 6392.299603] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6392.307512] Lustre: Skipped 1 previous similar message [ 6393.311218] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6393.513450] Lustre: server umount lustre-OST0000 complete [ 6396.896417] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6396.906606] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6396.923693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6402.891374] Lustre: server umount lustre-MDT0000 complete [ 6406.978460] LustreError: 130743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781243287 with bad export cookie 8777476071714449913 [ 6406.987377] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6407.315846] Lustre: server umount lustre-MDT0001 complete [ 6417.492875] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 6420.515530] Key type lgssc unregistered [ 6420.762520] LNet: 141523:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6420.768974] LNetError: 141523:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6420.783985] LNet: Removed LNI 192.168.201.116@tcp [ 6421.605342] Key type .llcrypt unregistered [ 6421.608826] Key type ._llcrypt unregistered [ 6440.125458] Key type ._llcrypt registered [ 6440.126990] Key type .llcrypt registered [ 6440.247713] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 6452.466846] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 6453.208442] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6453.333061] alg: No test for adler32 (adler32-zlib) [ 6454.360608] Lustre: Lustre: Build Version: 2.17.53_79_gff07dcf [ 6454.569693] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 6456.215230] Key type lgssc registered [ 6457.206900] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6489.909399] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6489.947929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6491.248499] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6491.313783] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6491.385267] Lustre: lustre-MDT0000: new disk, initializing [ 6491.490177] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6491.509799] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6495.159163] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6504.592374] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6504.709703] Lustre: 144667:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6504.782540] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6504.787780] Lustre: Skipped 1 previous similar message [ 6504.870856] Lustre: lustre-MDT0001: new disk, initializing [ 6504.955451] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6504.983669] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6504.992482] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6508.899255] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6515.593642] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6519.165458] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6525.949977] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6526.373342] Lustre: lustre-OST0000: new disk, initializing [ 6526.380199] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6526.392520] Lustre: 146410:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6526.542908] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6532.363722] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6536.763211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6536.775775] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6536.935070] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6539.100173] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6545.297648] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_module obdecho/obdecho [ 6547.200349] LustreError: 147283:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6547.204375] LustreError: 147283:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6549.011867] LustreError: 147380:0:(echo_client.c:1883:echo_md_handler()) Can not resolve the path : rc = -2 [ 6561.250828] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6561.256875] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6561.274069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6562.787259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6562.793968] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6562.814338] Lustre: Skipped 1 previous similar message [ 6565.848053] Lustre: server umount lustre-MDT0000 complete [ 6569.286876] LustreError: 144662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781243449 with bad export cookie 1566314471527289836 [ 6569.295884] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6569.567128] Lustre: server umount lustre-MDT0001 complete [ 6583.681541] Lustre: server umount lustre-OST0000 complete [ 6604.513663] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 6612.845686] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 6650.585785] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6650.853206] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6650.890636] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6650.947381] Lustre: lustre-MDT0000: new disk, initializing [ 6651.018139] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6651.033182] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6655.230917] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6664.690755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6664.804945] Lustre: 151061:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6664.885146] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6664.888897] Lustre: Skipped 1 previous similar message [ 6664.951910] Lustre: lustre-MDT0001: new disk, initializing [ 6665.047858] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6665.093159] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6665.113895] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6669.063564] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6675.123695] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6678.187706] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6683.863148] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6684.149740] Lustre: lustre-OST0000: new disk, initializing [ 6684.154591] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6684.162824] Lustre: 152802:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6684.221969] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6689.772516] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6691.412383] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6691.423662] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6691.498372] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6696.899240] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6700.776323] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6701.014243] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6705.064082] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6705.288936] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6707.174785] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6707.178952] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6707.189402] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6707.197308] Lustre: Skipped 1 previous similar message [ 6707.213437] Lustre: Skipped 3 previous similar messages [ 6712.289865] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6712.302459] Lustre: Skipped 1 previous similar message [ 6713.283312] Lustre: server umount lustre-OST0000 complete [ 6717.409396] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6717.418309] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6717.421806] Lustre: Skipped 1 previous similar message [ 6717.437505] Lustre: Skipped 1 previous similar message [ 6723.450056] Lustre: server umount lustre-MDT0000 complete [ 6727.253652] LustreError: 151054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781243607 with bad export cookie 1566314471527290802 [ 6727.263793] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6727.265781] LustreError: 151054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 6727.514227] Lustre: server umount lustre-MDT0001 complete [ 6730.737952] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 01:53:30 (1781243610) [ 6732.245682] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6733.986626] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 01:53:33 (1781243613) [ 6741.467254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6741.859891] LustreError: 155068:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6741.881101] LustreError: 155068:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6741.950640] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6745.942518] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6747.107331] LustreError: 155069:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6752.227377] LustreError: 155068:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6754.616611] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6758.446128] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6763.383132] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6766.719341] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6773.171901] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6778.590300] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6783.058455] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6795.744895] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6795.756453] LustreError: Skipped 1 previous similar message [ 6795.758401] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6795.779442] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6795.786900] Lustre: Skipped 2 previous similar messages [ 6800.001711] Lustre: server umount lustre-MDT0000 complete [ 6802.420384] LustreError: 155063:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6802.434208] LustreError: 155063:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6803.116138] LustreError: 156685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781243683 with bad export cookie 1566314471527291684 [ 6803.120184] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6803.130247] LustreError: 156685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6807.521180] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6807.522353] LustreError: 155064:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6807.540384] Lustre: Skipped 2 previous similar messages [ 6807.561388] LustreError: 155064:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6809.396749] Lustre: server umount lustre-MDT0001 complete [ 6815.048858] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6826.194246] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6826.352483] LustreError: 158568:0:(llog.c:1646:llog_backup()) MGC192.168.201.116@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6826.359306] Lustre: 158568:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.116@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6833.121975] LustreError: 158573:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6833.131467] LustreError: 158573:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6850.222170] LustreError: 158573:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6850.268843] LustreError: 158573:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 14 previous similar messages [ 6883.809765] LustreError: 156673:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6883.825298] LustreError: 156673:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 25 previous similar messages [ 6887.392072] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6887.397161] Lustre: 158568:0:(mgc_request.c:1901:mgc_process_log()) MGC192.168.201.116@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6928.391548] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6928.395081] Lustre: Skipped 2 previous similar messages [ 6933.498430] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6934.073275] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6943.392213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6947.819593] LustreError: 158573:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6947.859838] LustreError: 158573:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 43 previous similar messages [ 6953.451282] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6953.848491] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6961.349624] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6963.051535] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6967.566661] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6969.498271] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6989.284821] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6989.289947] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6989.296179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6989.300264] Lustre: Skipped 1 previous similar message [ 6989.308285] Lustre: Skipped 8 previous similar messages [ 6991.358872] Lustre: server umount lustre-MDT0000 complete [ 6995.652301] LustreError: 155048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781243875 with bad export cookie 1566314471527298138 [ 6995.652741] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6995.661701] LustreError: 155048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6996.074595] Lustre: server umount lustre-MDT0001 complete [ 7011.600044] Lustre: server umount lustre-OST0000 complete [ 7039.505988] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 7053.294899] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 7104.038156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7104.542302] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7104.585813] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7104.668308] Lustre: lustre-MDT0000: new disk, initializing [ 7104.768580] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7104.771345] Lustre: Skipped 1 previous similar message [ 7104.793043] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7110.285480] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7122.839356] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7122.965306] Lustre: 163418:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7123.152991] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7128.079383] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7138.503917] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7143.995454] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7153.312264] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7153.554488] Lustre: lustre-OST0000: new disk, initializing [ 7153.556712] Lustre: Skipped 1 previous similar message [ 7153.559882] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7153.562959] Lustre: Skipped 2 previous similar messages [ 7153.566582] Lustre: 165158:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7162.092786] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7162.957749] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7162.972981] Lustre: Skipped 1 previous similar message [ 7162.980395] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7163.102729] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7171.637984] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7175.916967] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7176.159379] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7182.238593] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7182.436514] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7188.964221] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7188.967981] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7188.979955] Lustre: Skipped 2 previous similar messages [ 7189.001724] Lustre: Skipped 3 previous similar messages [ 7191.400904] Lustre: server umount lustre-OST0000 complete [ 7202.325520] Lustre: server umount lustre-MDT0000 complete [ 7204.325478] LustreError: 163427:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7204.336463] LustreError: 163427:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 7206.910454] LustreError: 163409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781244087 with bad export cookie 1566314471527314973 [ 7206.912895] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7206.924023] LustreError: 163409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7207.409771] Lustre: server umount lustre-MDT0001 complete [ 7218.249452] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 02:01:37 (1781244097) [ 7226.405342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7231.887826] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7243.596809] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7244.046343] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7244.058491] Lustre: Skipped 3 previous similar messages [ 7249.772368] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7256.957257] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7261.381545] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7270.113748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7278.678944] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7287.010386] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7303.560477] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 02:03:02 (1781244182) [ 7308.769588] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7308.776452] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7308.794597] Lustre: Skipped 2 previous similar messages [ 7308.804783] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7308.807349] Lustre: Skipped 2 previous similar messages [ 7314.318200] Lustre: server umount lustre-OST0000 complete [ 7333.343163] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7333.597968] Lustre: server umount lustre-MDT0000 complete [ 7338.606375] LustreError: 167416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781244218 with bad export cookie 1566314471527315855 [ 7338.614912] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7338.621593] LustreError: 167416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7339.164763] Lustre: server umount lustre-MDT0001 complete [ 7354.526514] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 7356.974458] Key type lgssc unregistered [ 7357.264780] LNet: 170810:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7357.277947] LNetError: 170810:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7357.300268] LNet: Removed LNI 192.168.201.116@tcp [ 7358.305531] Key type .llcrypt unregistered [ 7358.308119] Key type ._llcrypt unregistered [ 7379.607724] Key type ._llcrypt registered [ 7379.612110] Key type .llcrypt registered [ 7379.783305] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 7381.297176] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7381.430353] alg: No test for adler32 (adler32-zlib) [ 7382.564298] Lustre: Lustre: Build Version: 2.17.53_79_gff07dcf [ 7382.960671] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 7384.679823] Key type lgssc registered [ 7386.047688] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7395.830694] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7395.862692] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7411.423379] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7416.543243] LustreError: 172046:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.116@tcp: failed processing log, type 4: rc = -110 [ 7421.663456] LustreError: 172046:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7421.679385] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7421.689537] LustreError: MGC192.168.201.116@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7421.707056] LustreError: 172046:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7421.720986] LustreError: 172046:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -22 [ 7421.736126] LustreError: 172046:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 7421.839644] Lustre: server umount lustre-OST0000 complete [ 7421.843086] LustreError: 172046:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7431.867647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7433.394342] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7438.544282] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7442.921112] LustreError: 172546:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7442.935810] LustreError: 172546:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7448.045502] LustreError: 172545:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7449.680845] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7450.194202] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7455.655841] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7462.807141] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7467.118514] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7475.099944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7475.368902] LustreError: 174134:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7475.391032] Lustre: 174134:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.201.116@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7475.574126] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7481.047346] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7486.747249] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7491.049281] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7491.058329] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7491.086657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7494.115288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7494.119117] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7494.135585] Lustre: Skipped 1 previous similar message [ 7494.155920] Lustre: Skipped 1 previous similar message [ 7495.862567] Lustre: server umount lustre-MDT0000 complete [ 7499.233251] LustreError: 174452:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7499.246886] LustreError: 174452:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7499.649964] LustreError: 172527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781244379 with bad export cookie 10402905791352261163 [ 7499.658619] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7499.859210] Lustre: server umount lustre-MDT0001 complete [ 7515.407510] Lustre: server umount lustre-OST0000 complete [ 7527.025870] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 7539.280797] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 7586.465182] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7586.688565] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7586.713997] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7586.794220] Lustre: lustre-MDT0000: new disk, initializing [ 7586.889095] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7586.905344] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7592.612480] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7607.023781] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7607.234963] Lustre: 177517:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7607.317781] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7607.323682] Lustre: Skipped 1 previous similar message [ 7607.434526] Lustre: lustre-MDT0001: new disk, initializing [ 7607.585983] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7607.620401] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7607.630078] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7614.106989] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7623.724705] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7629.390312] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7637.624451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7637.891212] Lustre: lustre-OST0000: new disk, initializing [ 7637.893612] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7637.901189] Lustre: 179259:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7637.998811] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7644.967793] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7645.274551] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7645.282873] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7645.466434] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7653.179494] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7658.609343] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7658.867494] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7665.454714] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7665.684565] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7671.264434] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7671.276160] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7671.286372] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7671.303578] Lustre: Skipped 1 previous similar message [ 7671.313391] Lustre: Skipped 1 previous similar message [ 7674.216797] Lustre: server umount lustre-OST0000 complete [ 7679.457063] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7679.467136] LustreError: Skipped 1 previous similar message [ 7679.468833] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7679.493934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7679.506421] Lustre: Skipped 1 previous similar message [ 7685.067355] Lustre: server umount lustre-MDT0000 complete [ 7686.624470] LustreError: 177529:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7686.644128] LustreError: 177529:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7689.398363] LustreError: 177509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781244569 with bad export cookie 10402905791352261884 [ 7689.414031] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7689.420981] LustreError: 177509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7690.126521] Lustre: server umount lustre-MDT0001 complete [ 7702.066528] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 02:09:41 (1781244581) [ 7722.876420] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 7726.170562] Key type lgssc unregistered [ 7726.580594] LNet: 182042:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7726.604417] LNetError: 182042:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7726.617050] LNet: Removed LNI 192.168.201.116@tcp [ 7727.763831] Key type .llcrypt unregistered [ 7727.767044] Key type ._llcrypt unregistered [ 7749.944303] Key type ._llcrypt registered [ 7749.949527] Key type .llcrypt registered [ 7750.090746] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 7751.394045] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7751.562920] alg: No test for adler32 (adler32-zlib) [ 7752.611850] Lustre: Lustre: Build Version: 2.17.53_79_gff07dcf [ 7752.827435] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 7754.511232] Key type lgssc registered [ 7755.724234] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7765.788338] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7765.833044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7767.729642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7772.881504] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7777.267246] LustreError: 183249:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7777.291266] LustreError: 183249:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7782.372884] LustreError: 183248:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7782.785770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7783.227121] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7788.185947] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7795.066923] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7800.688568] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7810.679603] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7811.031329] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7811.035315] LustreError: 184887:0:(llog.c:1687:llog_backup()) MGC192.168.201.116@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7811.048258] Lustre: 184887:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.201.116@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7811.175475] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7820.103340] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7827.274239] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7834.099220] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7834.113417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7834.123939] Lustre: Skipped 2 previous similar messages [ 7839.201860] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7839.213129] Lustre: Skipped 2 previous similar messages [ 7844.319183] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7844.344689] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7844.352516] Lustre: Skipped 2 previous similar messages [ 7844.571475] Lustre: server umount lustre-MDT0000 complete [ 7848.977058] LustreError: 183228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781244729 with bad export cookie 14992993298374830392 [ 7848.988520] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7848.997519] LustreError: 183228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7849.518416] Lustre: server umount lustre-MDT0001 complete [ 7864.689904] Lustre: server umount lustre-OST0000 complete [ 7876.478581] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 7890.495891] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 7929.923372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7930.121909] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7930.142689] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7930.195649] Lustre: lustre-MDT0000: new disk, initializing [ 7930.256348] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7930.268631] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7935.907992] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7949.575643] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7949.654321] Lustre: 188287:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7949.684407] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7949.686130] Lustre: Skipped 1 previous similar message [ 7949.688626] LustreError: 189053:0:(llog.c:1687:llog_backup()) MGC192.168.201.116@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7949.693209] LustreError: 189053:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 7949.699920] Lustre: 189053:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.116@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7949.712126] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7949.714539] Lustre: Skipped 1 previous similar message [ 7949.762652] Lustre: lustre-MDT0001: new disk, initializing [ 7949.833742] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7949.895668] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7949.900933] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7954.567425] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7963.443375] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7967.919076] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7976.025886] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7976.375873] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7976.378616] Lustre: Skipped 1 previous similar message [ 7976.380965] LustreError: 190005:0:(llog.c:1687:llog_backup()) MGC192.168.201.116@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7976.392850] LustreError: 190005:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 7976.396923] Lustre: 190005:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.116@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7976.408472] Lustre: 190005:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7976.447373] Lustre: lustre-OST0000: new disk, initializing [ 7976.450455] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7976.455864] Lustre: 190031:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7976.616657] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7984.424439] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7985.821483] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7985.841481] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7985.981680] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7993.331364] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7998.411913] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7998.794101] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8004.242983] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8004.442843] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8006.635387] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8006.649188] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8006.660339] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8006.664475] Lustre: Skipped 3 previous similar messages [ 8006.687072] LustreError: Skipped 1 previous similar message [ 8011.750598] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8011.753232] Lustre: Skipped 2 previous similar messages [ 8012.613515] Lustre: server umount lustre-OST0000 complete [ 8021.472649] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8021.479063] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8021.494860] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8023.398928] Lustre: server umount lustre-MDT0000 complete [ 8027.107871] LustreError: 189379:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8027.131481] LustreError: 189379:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8027.917225] LustreError: 188279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781244908 with bad export cookie 14992993298374831113 [ 8027.919505] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8027.942219] LustreError: 188279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8028.347892] Lustre: server umount lustre-MDT0001 complete [ 8038.776578] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 02:15:17 (1781244917) [ 8058.306497] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 8069.568425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8069.954620] LustreError: 192995:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8070.026115] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8075.236294] LustreError: 192996:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8075.260040] LustreError: 192996:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8075.577479] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 8085.067375] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8091.672476] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 8095.001354] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8103.313343] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8103.799346] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8103.806230] Lustre: Skipped 1 previous similar message [ 8111.766870] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 8121.984584] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8122.159957] Lustre: lustre-OST0001: new disk, initializing [ 8122.163610] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8122.170701] Lustre: 195253:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8128.676832] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8128.688296] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8128.798308] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8129.762449] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 8142.541581] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8153.111570] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8155.947511] Lustre: DEBUG MARKER: ADD OST3 [ 8162.709337] loop: module loaded [ 8162.828303] loop0: detected capacity change from 0 to 204800000 [ 8163.191315] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8163.207716] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8163.217339] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8170.875931] loop0: detected capacity change from 0 to 204800000 [ 8171.447598] Lustre: DEBUG MARKER: STOP OST3 [ 8178.753408] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8179.172135] Lustre: lustre-OST0003: new disk, initializing [ 8179.175596] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8179.183771] Lustre: 197277:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8179.372518] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8179.379798] Lustre: Skipped 1 previous similar message [ 8180.656195] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8180.781102] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8180.794592] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8180.842427] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8182.646516] Lustre: Failing over lustre-OST0003 [ 8182.687219] LustreError: 197395:0:(osp_precreate.c:1281:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8182.793350] Lustre: server umount lustre-OST0003 complete [ 8185.551833] LustreError: 195544:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.201.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8185.567640] LustreError: 195544:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8185.840180] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8185.857056] Lustre: Skipped 1 previous similar message [ 8185.872462] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8186.766353] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 8196.482786] Lustre: DEBUG MARKER: START OST3 again [ 8205.040801] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8205.664921] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8205.994691] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8207.108762] Lustre: lustre-OST0003: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 8207.109114] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8213.080650] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 8236.507628] Lustre: Failing over lustre-OST0003 [ 8236.528447] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8236.540387] Lustre: Skipped 1 previous similar message [ 8236.544678] Lustre: lustre-OST0003: Not available for connect from 0@lo (stopping) [ 8236.553157] Lustre: Skipped 2 previous similar messages [ 8236.615856] Lustre: server umount lustre-OST0003 complete [ 8241.642510] LustreError: 194447:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8241.672383] LustreError: 194447:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [ 8259.561559] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8259.576192] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8259.594885] Lustre: Skipped 1 previous similar message [ 8264.083253] Lustre: server umount lustre-MDT0000 complete [ 8268.638496] LustreError: 192976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781245148 with bad export cookie 14992993298374831995 [ 8268.645970] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8268.656956] LustreError: 192976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8272.361266] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8272.369517] Lustre: Skipped 6 previous similar messages [ 8275.337671] Lustre: server umount lustre-MDT0001 complete [ 8280.196919] Lustre: server umount lustre-OST0000 complete [ 8285.437182] Lustre: server umount lustre-OST0001 complete [ 8293.469453] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 8301.895483] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 8345.945962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8346.406380] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8346.431915] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8346.508275] Lustre: lustre-MDT0000: new disk, initializing [ 8346.672970] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8346.690133] Lustre: Skipped 1 previous similar message [ 8346.744327] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8351.723888] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 8363.038296] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8363.179291] Lustre: 202633:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8363.281857] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8363.287985] Lustre: Skipped 1 previous similar message [ 8363.405713] Lustre: lustre-MDT0001: new disk, initializing [ 8363.603483] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8363.616779] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8367.207378] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 8375.654994] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8380.688882] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8390.710568] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8390.898158] Lustre: 204376:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8392.662921] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8392.817101] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8396.943702] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 8406.277285] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8412.013476] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8412.524534] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8418.234086] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8418.569427] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8423.401401] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8423.401696] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8423.404758] LustreError: Skipped 1 previous similar message [ 8423.409613] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8423.432672] Lustre: Skipped 6 previous similar messages [ 8423.454730] Lustre: Skipped 3 previous similar messages [ 8427.342589] Lustre: server umount lustre-OST0000 complete [ 8439.459645] Lustre: server umount lustre-MDT0000 complete [ 8442.851183] LustreError: 202644:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8442.888134] LustreError: 202644:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 17 previous similar messages [ 8445.004921] LustreError: 202625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781245325 with bad export cookie 14992993298374834291 [ 8445.019136] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8445.026332] LustreError: 202625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8445.465815] Lustre: server umount lustre-MDT0001 complete [ 8449.425284] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 02:22:09 (1781245329) [ 8467.509620] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 8479.200749] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 8519.623328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8519.914919] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8520.007212] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8520.012401] Lustre: Skipped 1 previous similar message [ 8520.095931] Lustre: lustre-MDT0000: new disk, initializing [ 8520.098343] Lustre: Skipped 1 previous similar message [ 8520.174525] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8520.182503] Lustre: Skipped 1 previous similar message [ 8525.121579] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 8536.884830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8537.045673] Lustre: 208805:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8537.401265] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8542.331312] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 8550.612821] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8554.894592] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8562.268908] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8562.695662] Lustre: 210551:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8565.035996] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8569.495445] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 8577.746858] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8582.262776] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8582.455377] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8586.868853] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8587.070823] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8590.826929] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8590.828665] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8590.837205] LustreError: Skipped 1 previous similar message [ 8590.840888] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8590.865430] Lustre: Skipped 3 previous similar messages [ 8590.869054] Lustre: Skipped 4 previous similar messages [ 8603.103618] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8603.337502] Lustre: server umount lustre-OST0000 complete [ 8609.256934] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8613.527210] Lustre: server umount lustre-MDT0000 complete [ 8616.417615] LustreError: 208812:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8616.452354] LustreError: 208812:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8619.562414] LustreError: 208799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781245499 with bad export cookie 14992993298374835166 [ 8619.591208] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8619.592801] LustreError: 208799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8620.607373] Lustre: server umount lustre-MDT0001 complete [ 8632.098000] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8632.901685] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8632.904928] Lustre: Skipped 5 previous similar messages [ 8637.907506] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 8648.232968] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8654.805321] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 8662.383427] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8667.201555] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8675.595609] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8682.314142] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 8688.857457] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8700.907346] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8700.916181] Lustre: Skipped 3 previous similar messages [ 8701.684822] Lustre: server umount lustre-OST0000 complete [ 8711.493974] Lustre: server umount lustre-MDT0000 complete [ 8714.959122] LustreError: 212604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781245595 with bad export cookie 14992993298374836055 [ 8714.979189] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8715.300218] Lustre: server umount lustre-MDT0001 complete [ 8829.227668] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 8831.996046] Key type lgssc unregistered [ 8832.243824] LNet: 215937:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8832.254404] LNetError: 215937:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8832.273048] LNet: Removed LNI 192.168.201.116@tcp [ 8833.029193] Key type .llcrypt unregistered [ 8833.030861] Key type ._llcrypt unregistered [ 8868.264266] Key type ._llcrypt registered [ 8868.267368] Key type .llcrypt registered [ 8868.613457] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 8886.211337] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 8887.953378] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8887.978984] alg: No test for adler32 (adler32-zlib) [ 8889.056421] Lustre: Lustre: Build Version: 2.17.53_79_gff07dcf [ 8889.357079] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 8891.049864] Key type lgssc registered [ 8892.158485] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8931.376719] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8931.406691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8932.725389] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8932.763028] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8932.840961] Lustre: lustre-MDT0000: new disk, initializing [ 8932.972196] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8932.987294] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8937.237355] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 8950.048132] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8950.287961] Lustre: 219259:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8950.362865] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8950.368797] Lustre: Skipped 1 previous similar message [ 8950.504197] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 8950.595564] Lustre: lustre-MDT0001: new disk, initializing [ 8950.825249] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8950.967165] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8950.999388] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8955.711861] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 8965.049911] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8969.924112] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8977.405556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8977.679981] Lustre: lustre-OST0000: new disk, initializing [ 8977.684049] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8977.691063] Lustre: 221001:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8977.800518] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8984.261768] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 8987.251964] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8987.265873] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8987.432183] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8992.803696] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8997.981796] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8998.278649] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9002.434442] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9002.653602] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9008.097289] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9008.106940] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9008.111160] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9008.119901] Lustre: Skipped 1 previous similar message [ 9013.221221] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9013.230949] Lustre: Skipped 2 previous similar messages [ 9018.348728] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9018.354165] Lustre: Skipped 1 previous similar message [ 9018.847195] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9019.060517] Lustre: server umount lustre-OST0000 complete [ 9023.458040] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9023.477218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9023.486691] Lustre: Skipped 1 previous similar message [ 9029.862319] Lustre: server umount lustre-MDT0000 complete [ 9034.676589] LustreError: 219249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781245914 with bad export cookie 16864082690137539859 [ 9034.692526] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9034.697307] LustreError: 219249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9035.192847] Lustre: server umount lustre-MDT0001 complete [ 9054.982271] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 9064.823766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9065.206829] LustreError: 223776:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9065.219314] LustreError: 223776:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9065.255574] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9068.591734] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 9070.563388] LustreError: 223777:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9075.685766] LustreError: 223776:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9079.350879] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9079.902948] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9085.442651] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 9088.891649] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9094.492314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9094.756874] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9099.157857] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 9108.231210] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9108.404961] Lustre: lustre-OST0001: new disk, initializing [ 9108.408120] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9108.412330] Lustre: 226032:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9110.163711] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9110.169870] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9110.280433] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9115.907820] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 9127.097592] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9137.900222] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9153.620470] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 02:33:52 (1781246032) [ 9157.611436] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9157.623625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9157.645213] Lustre: Skipped 4 previous similar messages [ 9157.666111] Lustre: Skipped 5 previous similar messages [ 9163.560409] Lustre: server umount lustre-MDT0000 complete [ 9167.857963] LustreError: 223777:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9167.937735] LustreError: 223777:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9171.282959] LustreError: 223758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781246051 with bad export cookie 16864082690137540741 [ 9171.334633] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9171.359953] LustreError: 223758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9172.582113] Lustre: server umount lustre-MDT0001 complete [ 9188.570761] Lustre: server umount lustre-OST0000 complete [ 9189.151111] Lustre: 217481:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781246053/real 1781246053] req@ffff8ff487728b40 x1867771563592960/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781246069 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9189.186617] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9193.091518] Lustre: server umount lustre-OST0001 complete [ 9201.127567] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 9212.610700] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 9257.111886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9257.302994] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9257.350920] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9257.442857] Lustre: lustre-MDT0000: new disk, initializing [ 9257.557279] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9257.559444] Lustre: Skipped 1 previous similar message [ 9257.571586] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9263.014456] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 9276.694259] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9276.859461] Lustre: 229925:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9276.921673] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9276.925122] Lustre: Skipped 1 previous similar message [ 9277.007983] Lustre: lustre-MDT0001: new disk, initializing [ 9277.197949] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9277.225736] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9281.823977] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 9292.276373] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9297.143612] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9305.755965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9306.205129] Lustre: lustre-OST0000: new disk, initializing [ 9306.210313] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9306.218687] Lustre: 231671:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9306.301553] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9306.305484] Lustre: Skipped 1 previous similar message [ 9308.222151] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9308.246324] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9308.381595] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9313.172371] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 9321.362823] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9326.961335] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9327.270466] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9334.473435] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9334.985787] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9339.363937] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9339.383855] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9339.386920] LustreError: Skipped 1 previous similar message [ 9339.410612] Lustre: Skipped 1 previous similar message [ 9339.411864] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9339.424460] Lustre: Skipped 5 previous similar messages [ 9344.556353] LustreError: 231653:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9344.914794] Lustre: server umount lustre-OST0000 complete [ 9349.605637] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9354.212192] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9354.225894] LustreError: Skipped 1 previous similar message [ 9355.684896] Lustre: server umount lustre-MDT0000 complete [ 9359.775603] LustreError: 229920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781246240 with bad export cookie 16864082690137542876 [ 9359.782096] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9359.789590] LustreError: 229920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9360.491383] Lustre: server umount lustre-MDT0001 complete [ 9369.197464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9369.718876] LustreError: 233744:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9369.748046] LustreError: 233744:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9374.849328] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 9384.385400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9384.953789] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9384.963192] Lustre: Skipped 1 previous similar message [ 9390.317431] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 9396.674724] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9401.022427] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9407.836273] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9413.865490] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 9420.155577] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9568.233659] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9568.234745] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9568.242166] Lustre: Skipped 2 previous similar messages [ 9568.242655] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9568.242660] Lustre: Skipped 3 previous similar messages [ 9571.470365] Lustre: server umount lustre-OST0000 complete [ 9583.660343] Lustre: server umount lustre-MDT0000 complete [ 9588.714078] LustreError: 233741:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9588.732091] LustreError: 233741:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 9590.370537] LustreError: 233725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781246470 with bad export cookie 16864082690137543758 [ 9590.379020] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9590.384499] LustreError: 233725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9591.169875] Lustre: server umount lustre-MDT0001 complete [ 9689.741798] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 9692.314906] Key type lgssc unregistered [ 9692.525776] LNet: 236991:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9692.531975] LNetError: 236991:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9693.605288] LNet: Removed LNI 192.168.201.116@tcp [ 9694.584255] Key type .llcrypt unregistered [ 9694.586100] Key type ._llcrypt unregistered [ 9732.030187] Key type ._llcrypt registered [ 9732.037579] Key type .llcrypt registered [ 9732.226553] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 9752.781920] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 9754.328543] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9754.373980] alg: No test for adler32 (adler32-zlib) [ 9755.458438] Lustre: Lustre: Build Version: 2.17.53_79_gff07dcf [ 9755.614442] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 9757.247567] Key type lgssc registered [ 9758.124267] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9811.537877] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 9827.801879] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9827.874632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9829.277262] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9829.328514] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9829.422560] Lustre: lustre-MDT0000: new disk, initializing [ 9829.532107] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9829.559509] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9834.887585] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 9847.212519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9847.403712] Lustre: 241036:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9847.494901] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9847.509191] Lustre: Skipped 1 previous similar message [ 9847.631249] Lustre: lustre-MDT0001: new disk, initializing [ 9847.729547] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9847.765655] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9847.772173] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9852.226221] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 9857.527266] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9865.011498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9865.374437] Lustre: lustre-OST0000: new disk, initializing [ 9865.378912] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9865.386652] Lustre: 242635:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9865.486933] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9870.854264] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9870.867818] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9870.895372] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 9871.549790] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 9883.924929] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9884.229088] Lustre: lustre-OST0001: new disk, initializing [ 9884.248139] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9884.261662] Lustre: 243489:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9884.460474] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9892.150716] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 9893.509164] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9893.514920] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9893.645913] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9905.966206] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9914.412627] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9929.700444] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9929.707405] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9929.734729] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9932.782969] Lustre: server umount lustre-MDT0000 complete [ 9934.824200] LustreError: 243048:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9934.855409] LustreError: 243048:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 9937.444571] LustreError: 241028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781246817 with bad export cookie 10260493050664151964 [ 9937.457146] LustreError: 241028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9937.458271] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9938.008983] Lustre: server umount lustre-MDT0001 complete [ 9954.383984] Lustre: server umount lustre-OST0000 complete [ 9956.319306] Lustre: 238557:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781246820/real 1781246820] req@ffff8ff38362a1c0 x1867772471621760/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781246836 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9956.366167] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9956.372800] Lustre: Skipped 3 previous similar messages [ 9958.694133] Lustre: server umount lustre-OST0001 complete [ 9974.768698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9986.426928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9997.818965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10012.497719] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10036.553985] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [10047.803366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10047.817916] Lustre: lustre-MDT0000: reset Object Index mappings [10048.058604] LustreError: 248465:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10048.096846] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10053.011916] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [10053.609228] LustreError: 248466:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10058.721343] LustreError: 248465:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10061.892779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10061.908560] Lustre: lustre-MDT0001: reset Object Index mappings [10062.361586] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10067.005855] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [10070.827515] Lustre: 249575:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10080.678738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10081.171709] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10087.346648] LustreError: 249928:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10088.670140] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [10093.626215] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [10098.665900] LustreError: 249929:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10098.684626] LustreError: 249929:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [10099.525766] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10104.836638] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10106.543856] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [10116.548436] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10121.578685] Lustre: 251410:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10127.903069] Lustre: server umount lustre-MDT0000 complete [10128.864618] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10128.879809] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10128.904873] LustreError: 248466:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10131.761058] LustreError: 248446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781247011 with bad export cookie 10260493050664158754 [10131.762377] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10131.773922] LustreError: 248446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10132.080481] Lustre: server umount lustre-MDT0001 complete [10146.427633] Lustre: server umount lustre-OST0000 complete [10160.747108] Lustre: server umount lustre-OST0001 complete [10183.845983] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 02:51:03 (1781247063) [10204.500662] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [10215.062384] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [10264.255820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10264.591196] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10264.620698] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10264.716540] Lustre: lustre-MDT0000: new disk, initializing [10264.853316] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10264.857437] Lustre: Skipped 1 previous similar message [10264.882538] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10269.765925] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [10282.519030] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10282.700657] Lustre: 255633:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10282.767632] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10282.777050] Lustre: Skipped 1 previous similar message [10282.859865] Lustre: lustre-MDT0001: new disk, initializing [10283.121205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10283.134459] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10289.889752] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [10299.776938] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10305.474126] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10313.610101] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10314.035768] Lustre: lustre-OST0000: new disk, initializing [10314.048688] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10314.064606] Lustre: 257374:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10315.463429] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10315.479380] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10315.637204] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10321.391701] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [10330.691537] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10336.211618] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10336.378992] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10341.229583] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10341.490633] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10346.477669] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10346.479162] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10346.493325] LustreError: Skipped 1 previous similar message [10346.501758] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10346.526212] Lustre: Skipped 4 previous similar messages [10346.551102] Lustre: Skipped 3 previous similar messages [10351.607695] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10351.616527] Lustre: Skipped 1 previous similar message [10356.705337] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10356.710088] Lustre: Skipped 1 previous similar message [10358.239244] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10358.413852] Lustre: server umount lustre-OST0000 complete [10361.824160] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10361.826322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10361.832239] Lustre: Skipped 1 previous similar message [10361.852258] Lustre: Skipped 1 previous similar message [10368.038339] Lustre: server umount lustre-MDT0000 complete [10372.066152] LustreError: 255645:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10372.090426] LustreError: 255645:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [10372.205643] LustreError: 255627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781247252 with bad export cookie 10260493050664160847 [10372.217092] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10372.454149] Lustre: server umount lustre-MDT0001 complete [10390.221157] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [10401.648541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10402.153935] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10402.156642] Lustre: Skipped 2 previous similar messages [10406.738437] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [10416.404746] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10421.104649] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [10424.742890] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10433.323430] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10441.715837] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [10451.190344] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10451.395897] Lustre: lustre-OST0001: new disk, initializing [10451.399771] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10451.405964] Lustre: 262404:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10452.729848] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10452.741813] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10452.838815] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10457.913559] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [10467.995386] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10471.347889] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10476.942151] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10477.117972] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10483.686470] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10483.689862] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10483.709652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10483.711548] Lustre: Skipped 3 previous similar messages [10489.785569] Lustre: server umount lustre-MDT0000 complete [10492.892877] LustreError: 260130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781247373 with bad export cookie 10260493050664161729 [10492.897522] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10492.903488] LustreError: 260130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [10493.312383] Lustre: server umount lustre-MDT0001 complete [10507.254239] Lustre: server umount lustre-OST0000 complete [10510.304179] Lustre: 238558:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781247374/real 1781247374] req@ffff8ff381e812c0 x1867772471782784/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781247390 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10510.346257] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10510.367750] Lustre: Skipped 3 previous similar messages [10511.140364] Lustre: server umount lustre-OST0001 complete [10527.823609] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [10536.415795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10536.720431] LustreError: 265423:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10536.734983] LustreError: 265423:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10536.804548] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10536.809946] Lustre: Skipped 3 previous similar messages [10540.580143] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [10548.174634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10551.673784] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [10554.246972] Lustre: 266527:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10560.108979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10561.594174] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10565.477691] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [10572.552619] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10573.827546] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10577.432241] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [10583.419659] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10586.302402] Lustre: 268372:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10589.890168] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10589.912222] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10596.545835] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 02:57:56 (1781247476) [10610.148474] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10610.154366] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10610.166985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10610.171242] Lustre: Skipped 5 previous similar messages [10616.156190] Lustre: server umount lustre-MDT0000 complete [10618.822684] LustreError: 265406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781247499 with bad export cookie 10260493050664168302 [10618.837183] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10618.837937] LustreError: 265406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10633.183166] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10633.507429] Lustre: server umount lustre-MDT0001 complete [10642.557583] Lustre: server umount lustre-OST0000 complete [10651.434406] Lustre: server umount lustre-OST0001 complete [10656.521829] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [10663.047993] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [10691.116187] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [10697.240780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10697.440989] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10697.461578] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10697.510783] Lustre: lustre-MDT0000: new disk, initializing [10697.582156] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10700.141975] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [10707.557322] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10707.663148] Lustre: 272553:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10707.859370] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10710.364269] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [10717.709417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10717.846459] Lustre: 274157:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10719.176345] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10721.638556] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [10729.324463] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10729.456331] Lustre: 275011:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10731.251628] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10733.823642] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [10741.614639] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10744.221554] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10744.225571] Lustre: Skipped 1 previous similar message [10752.288054] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 03:00:32 (1781247632) [10754.015870] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10754.021243] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10754.026817] Lustre: Skipped 5 previous similar messages [10754.028973] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10754.032668] Lustre: Skipped 10 previous similar messages [10759.483726] Lustre: server umount lustre-MDT0000 complete [10761.422407] LustreError: 273334:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781247641 with bad export cookie 10260493050664175960 [10761.423515] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10761.427364] LustreError: 273334:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10761.654791] Lustre: server umount lustre-MDT0001 complete [10774.259194] Lustre: server umount lustre-OST0000 complete [10787.209704] Lustre: server umount lustre-OST0001 complete [10792.201517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10792.514062] LustreError: 277250:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10792.524684] LustreError: 277250:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [10795.059463] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [10800.437646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10800.658511] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10800.661462] Lustre: Skipped 8 previous similar messages [10803.291913] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [10806.284829] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10808.553786] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10812.294451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10815.704931] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [10818.468542] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10837.289231] loop1: detected capacity change from 0 to 204800000 [10837.446741] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10837.450978] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10837.456211] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10840.512120] loop1: detected capacity change from 0 to 204800000 [10843.708725] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10843.771076] LustreError: 277238:0:(mgs_llog.c:673:mgs_find_or_make_fsdb_nolock()) fs18264: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10843.776535] LustreError: 277238:0:(mgs_llog.c:731:mgs_check_index()) Can't get db for fs18264 [10843.779094] LustreError: 277238:0:(mgs_handler.c:517:mgs_target_reg()) fs18264-MDT0001: Error checking target index: rc = -13 [10843.782731] LustreError: fs18264-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10843.786286] LustreError: 280644:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -13 [10843.789224] LustreError: 280644:0:(tgt_mount.c:2084:server_put_super()) no obd fs18264-MDT0001 [10843.791331] LustreError: 280644:0:(tgt_mount.c:116:server_deregister_mount()) fs18264-MDT0001 not registered [10843.804950] Lustre: server umount fs18264-MDT0001 complete [10843.807387] LustreError: 280644:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [10849.223575] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10849.296595] Lustre: Setting parameter fs18264-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs18264-MDT0001 [10849.332161] Lustre: srv-fs18264-MDT0001: No data found on store. Initialize space. [10849.334412] Lustre: Skipped 4 previous similar messages [10849.395286] Lustre: fs18264-MDT0001: new disk, initializing [10849.397635] Lustre: Skipped 3 previous similar messages [10849.499802] Lustre: srv-fs18264-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10849.508277] Lustre: 281111:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs18264-MDT0001-mdtlov: can't create new llog: rc = -115 [10849.514260] LustreError: 281111:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs18264-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10852.462447] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [10857.369860] Lustre: server umount fs18264-MDT0001 complete [10865.937828] loop1: detected capacity change from 0 to 204800000 [10869.515678] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10869.765563] Lustre: 282660:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs18264-MDT0001-mdtlov: can't create new llog: rc = -115 [10869.772935] LustreError: 282660:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs18264-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10872.407097] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [10875.713754] Lustre: server umount fs18264-MDT0001 complete [10885.317338] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.16@tcp (stopping) [10885.320644] Lustre: Skipped 4 previous similar messages [10886.629404] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10886.642095] Lustre: Skipped 4 previous similar messages [10889.484370] Lustre: server umount lustre-MDT0000 complete [10894.879653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10894.938855] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10897.758664] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [10900.465638] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10902.289563] loop1: detected capacity change from 0 to 204800000 [10905.514978] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10905.716196] Lustre: 285068:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs18264-MDT0001-mdtlov: can't create new llog: rc = -115 [10905.719916] LustreError: 285068:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs18264-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10908.076348] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [10910.691263] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10911.094548] Lustre: server umount fs18264-MDT0001 complete [10928.871512] Lustre: server umount lustre-OST0000 complete [10937.311165] Lustre: server umount lustre-MDT0000 complete [10939.194721] LustreError: 286641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781247819 with bad export cookie 10260493050664178641 [10939.195169] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10939.198691] LustreError: 286641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10939.342169] Lustre: server umount lustre-MDT0001 complete [10944.091296] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [10945.276916] Key type lgssc unregistered [10945.426427] LNet: 287374:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10945.431679] LNetError: 287374:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10945.441406] LNet: Removed LNI 192.168.201.116@tcp [10945.827975] Key type .llcrypt unregistered [10945.830528] Key type ._llcrypt unregistered [10955.688157] Key type ._llcrypt registered [10955.689337] Key type .llcrypt registered [10955.745802] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [10956.136376] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10956.166184] alg: No test for adler32 (adler32-zlib) [10957.091537] Lustre: Lustre: Build Version: 2.17.53_79_gff07dcf [10957.197708] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [10958.807286] Key type lgssc registered [10959.347081] Lustre: Echo OBD driver; http://www.lustre.org/ [10963.642817] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10963.662519] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10964.967554] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10966.841606] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [10970.933351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10971.158295] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10972.826638] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [10975.041593] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10976.850989] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10979.898718] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10980.113375] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10981.158611] LustreError: 290206:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10981.164353] LustreError: 290206:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10982.836650] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [10985.709618] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10986.466659] LustreError: 290347:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10990.566816] LustreError: 290207:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10990.576018] LustreError: 290207:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10995.684656] LustreError: 290206:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10995.691934] LustreError: 290206:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11000.809304] LustreError: 290206:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11000.815465] LustreError: 290206:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11001.464237] loop1: detected capacity change from 0 to 204800000 [11001.600837] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11001.610833] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11001.626103] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11004.567441] loop1: detected capacity change from 0 to 204800000 [11007.385781] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11007.498334] Lustre: fs21104-OST0000: new disk, initializing [11007.501100] Lustre: srv-fs21104-OST0000: No data found on store. Initialize space. [11007.505471] Lustre: 291998:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11007.571833] Lustre: fs21104-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11010.568368] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [11011.044433] LustreError: 290206:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11011.050855] LustreError: 290206:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [11014.928898] Lustre: server umount fs21104-OST0000 complete [11023.065879] loop1: detected capacity change from 0 to 204800000 [11028.142719] LustreError: 290347:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11028.153288] LustreError: 290347:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [11029.221563] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11029.385298] Lustre: fs21104-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11036.921347] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [11047.447133] loop2: detected capacity change from 0 to 204800000 [11047.969703] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11047.981756] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11048.003200] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11056.080881] loop2: detected capacity change from 0 to 204800000 [11062.244463] LustreError: 290208:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11062.274305] LustreError: 290208:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 19 previous similar messages [11065.029083] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11065.231475] LustreError: fs21104-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11065.250303] LustreError: fs21104-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11065.262874] LustreError: 295007:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -13 [11065.279322] LustreError: 295007:0:(tgt_mount.c:2084:server_put_super()) no obd fs21104-OST0001 [11065.290458] LustreError: 295007:0:(tgt_mount.c:116:server_deregister_mount()) fs21104-OST0001 not registered [11065.340702] Lustre: server umount fs21104-OST0001 complete [11065.349656] LustreError: 295007:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [11072.501936] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11077.615808] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11082.199736] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11082.478521] Lustre: fs21104-OST0001: new disk, initializing [11082.480562] Lustre: srv-fs21104-OST0001: No data found on store. Initialize space. [11082.487222] Lustre: 295558:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11082.591586] Lustre: fs21104-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11092.112379] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [11103.611893] Lustre: server umount fs21104-OST0000 complete [11113.938105] Lustre: server umount fs21104-OST0001 complete [11126.587505] loop1: detected capacity change from 0 to 204800000 [11128.823184] LustreError: 290347:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11128.832522] LustreError: 290347:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 38 previous similar messages [11132.481267] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11132.744943] Lustre: fs21104-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11138.640838] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [11144.824782] loop2: detected capacity change from 0 to 204800000 [11150.559630] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11155.910498] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [11164.657688] Lustre: server umount fs21104-OST0000 complete [11174.409853] Lustre: server umount fs21104-OST0001 complete [11190.244031] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11190.254428] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11190.269383] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11190.274122] Lustre: Skipped 1 previous similar message [11190.284934] Lustre: Skipped 1 previous similar message [11193.304044] Lustre: server umount lustre-OST0000 complete [11200.482040] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11200.485781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11200.488636] Lustre: Skipped 1 previous similar message [11200.502293] Lustre: Skipped 1 previous similar message [11203.775102] Lustre: server umount lustre-MDT0000 complete [11207.636921] LustreError: 288587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781248087 with bad export cookie 13961396673482285487 [11207.638280] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11207.646078] LustreError: 288587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11208.039973] Lustre: server umount lustre-MDT0001 complete [11217.405420] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [11219.887658] Key type lgssc unregistered [11220.206362] LNet: 301412:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11220.210631] LNetError: 301412:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11220.227714] LNet: Removed LNI 192.168.201.116@tcp [11220.933214] Key type .llcrypt unregistered [11220.935278] Key type ._llcrypt unregistered [11250.065844] Key type ._llcrypt registered [11250.069182] Key type .llcrypt registered [11250.179355] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [11264.296613] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [11265.468165] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11265.571094] alg: No test for adler32 (adler32-zlib) [11266.724520] Lustre: Lustre: Build Version: 2.17.53_79_gff07dcf [11266.994347] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [11268.631306] Key type lgssc registered [11269.673349] Lustre: Echo OBD driver; http://www.lustre.org/ [11303.356101] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11303.389728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11304.623051] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11304.651241] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11304.711296] Lustre: lustre-MDT0000: new disk, initializing [11304.774527] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11304.793943] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11308.996336] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [11318.598706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11318.710878] Lustre: 305238:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11318.768991] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11318.772157] Lustre: Skipped 1 previous similar message [11318.842860] Lustre: lustre-MDT0001: new disk, initializing [11318.953684] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11318.988053] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11318.995626] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11323.105898] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [11330.642226] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11333.858872] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11340.177794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11340.425153] Lustre: lustre-OST0000: new disk, initializing [11340.427991] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11340.433422] Lustre: 306980:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11340.518962] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11346.024869] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11346.028983] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11346.198860] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11347.285741] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [11354.317066] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11356.122174] Lustre: Mounted lustre-client [11356.957471] Lustre: Unmounted lustre-client [11366.374475] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11366.388197] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11366.394254] Lustre: Skipped 1 previous similar message [11366.403661] Lustre: Skipped 1 previous similar message [11371.498849] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11371.512190] Lustre: Skipped 1 previous similar message [11372.579685] Lustre: server umount lustre-OST0000 complete [11380.707668] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11380.712365] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11380.721819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11381.729674] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11384.391398] Lustre: server umount lustre-MDT0000 complete [11391.639831] LustreError: 306979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781248271 with bad export cookie 16469819035659850760 [11391.640672] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11391.647552] LustreError: 306979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11393.091634] Lustre: server umount lustre-MDT0001 complete [11409.507053] LNet: 1 local NIs in recovery (showing 1): 192.168.201.116@tcp [11413.292307] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [11417.430379] Key type lgssc unregistered [11417.825683] LNet: 309034:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11417.839266] LNetError: 309034:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11486.445129] LNet: Removed LNI 192.168.201.116@tcp [11487.650633] Key type .llcrypt unregistered [11487.652898] Key type ._llcrypt unregistered [11508.288840] Key type ._llcrypt registered [11508.293383] Key type .llcrypt registered [11508.395701] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [11524.119533] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [11525.978809] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11526.017317] alg: No test for adler32 (adler32-zlib) [11527.115731] Lustre: Lustre: Build Version: 2.17.53_79_gff07dcf [11527.451045] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [11529.182122] Key type lgssc registered [11530.612688] Lustre: Echo OBD driver; http://www.lustre.org/ [11564.920918] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 03:14:04 (1781248444) [11572.095990] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11572.129450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11573.440649] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11573.498754] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11573.599486] Lustre: lustre-MDT0000: new disk, initializing [11573.701158] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11573.734428] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11578.443077] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [11591.771919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11591.951639] Lustre: 312223:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11592.008536] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11592.012852] Lustre: Skipped 1 previous similar message [11592.142566] Lustre: lustre-MDT0001: new disk, initializing [11592.296457] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11592.364546] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11592.376358] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11597.349469] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [11606.762805] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11610.611703] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11617.920824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11618.306464] Lustre: lustre-OST0000: new disk, initializing [11618.309895] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11618.315529] Lustre: 313967:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11618.406663] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11625.114961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11625.123667] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11625.257762] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11625.266280] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [11633.762406] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11645.925529] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11645.927400] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11645.946563] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11649.230771] Lustre: server umount lustre-OST0000 complete [11656.163869] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11656.187075] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11656.191760] Lustre: Skipped 2 previous similar messages [11656.195536] Lustre: Skipped 1 previous similar message [11660.206690] Lustre: server umount lustre-MDT0000 complete [11665.246706] LustreError: 312213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781248545 with bad export cookie 3449713754378730206 [11665.258041] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11665.261291] LustreError: 312213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11665.707832] Lustre: server umount lustre-MDT0001 complete [11682.784882] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [11687.035613] Key type lgssc unregistered [11687.541415] LNet: 315597:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11687.554608] LNetError: 315597:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11687.595223] LNet: Removed LNI 192.168.201.116@tcp [11688.708941] Key type .llcrypt unregistered [11688.711479] Key type ._llcrypt unregistered [11724.062084] Key type ._llcrypt registered [11724.064898] Key type .llcrypt registered [11724.228333] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [11725.295846] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11725.317960] alg: No test for adler32 (adler32-zlib) [11726.463316] Lustre: Lustre: Build Version: 2.17.53_79_gff07dcf [11726.870202] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [11728.562079] Key type lgssc registered [11730.880883] Lustre: Echo OBD driver; http://www.lustre.org/ [11742.342374] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11742.361619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11744.075806] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11748.354755] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [11754.475427] LustreError: 317004:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11754.487384] LustreError: 317004:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11759.586279] LustreError: 317005:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11759.876390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11760.603308] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11766.434327] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [11774.260788] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11779.315419] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11789.683244] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11790.224470] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11800.973510] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [11809.739745] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11826.398304] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11829.074663] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11835.367236] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11835.383641] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11835.417328] Lustre: Skipped 1 previous similar message [11835.439639] Lustre: Skipped 2 previous similar messages [11837.646610] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.16@tcp (stopping) [11840.514473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11840.541440] Lustre: Skipped 2 previous similar messages [11840.817609] Lustre: server umount lustre-MDT0000 complete [11842.730851] LustreError: 316999:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11845.602863] LustreError: 317716:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11845.625288] LustreError: 317716:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11846.297831] LustreError: 316985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781248726 with bad export cookie 12715672264654000846 [11846.323551] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11847.887569] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.16@tcp (stopping) [11850.720740] LustreError: 316999:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11850.724487] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11850.751304] LustreError: 316999:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11850.778826] Lustre: Skipped 1 previous similar message [11852.904631] Lustre: server umount lustre-MDT0001 complete [11858.118877] Lustre: server umount lustre-OST0000 complete [11869.061595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11874.814723] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [11879.249083] Lustre: 320754:0:(mgs_llog.c:1798:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.116@tcp [11882.201769] Lustre: server umount MGS complete [11893.043570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11893.450159] LustreError: 321327:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11893.470764] LustreError: 321327:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11893.539341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11900.533321] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [11912.465791] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11912.889647] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11919.322190] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [11931.679091] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11932.182859] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11940.162251] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [11947.276237] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11949.282184] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [11953.802928] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11955.746282] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11961.840659] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [11961.851850] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11961.882652] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11961.892113] Lustre: Skipped 2 previous similar messages [11962.848298] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11963.373893] Lustre: server umount lustre-OST0000 complete [11968.204982] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11968.219891] Lustre: Skipped 1 previous similar message [11973.604910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11973.624045] Lustre: Skipped 4 previous similar messages [11974.595660] Lustre: server umount lustre-MDT0000 complete [11978.694581] LustreError: 321309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781248858 with bad export cookie 12715672264654001686 [11978.706334] LustreError: 321309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11978.708416] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11979.080670] Lustre: server umount lustre-MDT0001 complete [11992.026436] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [11995.129765] Key type lgssc unregistered [11995.512039] LNet: 324503:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11995.520771] LNetError: 324503:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11996.610047] LNet: Removed LNI 192.168.201.116@tcp [11997.616946] Key type .llcrypt unregistered [11997.619112] Key type ._llcrypt unregistered [12019.195555] Key type ._llcrypt registered [12019.204810] Key type .llcrypt registered [12019.414669] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [12035.235327] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [12036.215056] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12036.307271] alg: No test for adler32 (adler32-zlib) [12037.370205] Lustre: Lustre: Build Version: 2.17.53_79_gff07dcf [12037.808506] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [12039.559223] Key type lgssc registered [12041.095110] Lustre: Echo OBD driver; http://www.lustre.org/ [12078.546110] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12078.582191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12079.871621] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12079.908768] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12079.987801] Lustre: lustre-MDT0000: new disk, initializing [12080.104597] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12080.120197] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12084.606518] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [12096.601392] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12096.772731] Lustre: 327632:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12096.849463] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12096.857665] Lustre: Skipped 1 previous similar message [12097.017685] Lustre: lustre-MDT0001: new disk, initializing [12097.113162] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12097.159810] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12097.178032] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12102.346216] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [12110.172948] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12114.270639] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12121.773989] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12122.031869] Lustre: lustre-OST0000: new disk, initializing [12122.036600] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12122.045733] Lustre: 329373:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12122.148148] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12127.827815] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12127.838264] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12127.961946] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12128.669225] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [12135.598948] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12141.224990] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12141.431344] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12146.830492] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12147.020151] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12153.831523] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12153.832100] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12153.849857] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12153.858907] Lustre: Skipped 1 previous similar message [12153.877592] LustreError: Skipped 1 previous similar message [12155.381833] Lustre: server umount lustre-OST0000 complete [12158.945243] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12158.958574] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12158.961510] Lustre: Skipped 1 previous similar message [12163.552512] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12163.564690] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12163.591930] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12169.185740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12169.188428] Lustre: Skipped 2 previous similar messages [12173.279172] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12173.622768] Lustre: server umount lustre-MDT0000 complete [12177.885410] LustreError: 327623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781249058 with bad export cookie 14636183101864258057 [12177.886949] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12177.895133] LustreError: 327623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12178.199294] Lustre: server umount lustre-MDT0001 complete [12187.693471] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 03:24:27 (1781249067) [12204.837515] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [12214.582596] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [12252.362566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12252.635298] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12252.656967] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12252.707905] Lustre: lustre-MDT0000: new disk, initializing [12252.774853] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12252.793318] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12257.863302] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [12267.748889] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12267.853203] Lustre: 333953:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12267.898783] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12267.901405] Lustre: Skipped 1 previous similar message [12267.964838] Lustre: lustre-MDT0001: new disk, initializing [12268.036476] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12268.080629] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12268.096819] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12272.522802] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [12280.331316] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12284.844231] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12292.116611] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12292.405516] Lustre: lustre-OST0000: new disk, initializing [12292.408355] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12292.413235] Lustre: 335699:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12292.461957] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12298.822578] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12298.836811] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12298.915661] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [12298.952580] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12305.832724] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12340.195873] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12340.204098] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12340.218428] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12340.229360] Lustre: Skipped 1 previous similar message [12340.249604] Lustre: Skipped 1 previous similar message [12350.440635] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12350.443197] Lustre: Skipped 2 previous similar messages [12352.991530] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12353.276470] Lustre: server umount lustre-OST0000 complete [12365.282981] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12365.305681] LustreError: Skipped 1 previous similar message [12365.325952] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12368.156422] Lustre: server umount lustre-MDT0000 complete [12370.918223] LustreError: 334742:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12370.969992] LustreError: 334742:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [12375.233459] LustreError: 334732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781249255 with bad export cookie 14636183101864258946 [12375.237036] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12375.245187] LustreError: 334732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12376.596626] Lustre: server umount lustre-MDT0001 complete [12392.602750] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [12396.971432] Key type lgssc unregistered [12397.451431] LNet: 337742:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12397.465995] LNetError: 337742:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12397.493845] LNet: Removed LNI 192.168.201.116@tcp [12398.494244] Key type .llcrypt unregistered [12398.496587] Key type ._llcrypt unregistered [12497.290341] Key type ._llcrypt registered [12497.291836] Key type .llcrypt registered [12497.464779] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [12497.886801] Key type .llcrypt unregistered [12497.888325] Key type ._llcrypt unregistered [12594.352749] Key type ._llcrypt registered [12594.354494] Key type .llcrypt registered [12594.494795] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [12594.968299] Key type .llcrypt unregistered [12594.969601] Key type ._llcrypt unregistered [12632.237472] Key type ._llcrypt registered [12632.239039] Key type .llcrypt registered [12632.406271] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [12632.740969] Key type .llcrypt unregistered [12632.742682] Key type ._llcrypt unregistered [12669.438842] Key type ._llcrypt registered [12669.441472] Key type .llcrypt registered [12669.640118] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [12670.123390] Key type .llcrypt unregistered [12670.134723] Key type ._llcrypt unregistered [12756.915472] Key type ._llcrypt registered [12756.920638] Key type .llcrypt registered [12757.096309] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [12757.574472] Key type .llcrypt unregistered [12757.580106] Key type ._llcrypt unregistered [12793.303567] Key type ._llcrypt registered [12793.307258] Key type .llcrypt registered [12793.449907] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [12793.751359] Key type .llcrypt unregistered [12793.763539] Key type ._llcrypt unregistered [12827.427720] Key type ._llcrypt registered [12827.432701] Key type .llcrypt registered [12827.542689] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [12827.920604] Key type .llcrypt unregistered [12827.923326] Key type ._llcrypt unregistered [12860.344879] Key type ._llcrypt registered [12860.346510] Key type .llcrypt registered [12860.462476] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [12860.780618] Key type .llcrypt unregistered [12860.787342] Key type ._llcrypt unregistered [12895.777705] Key type ._llcrypt registered [12895.779844] Key type .llcrypt registered [12895.989302] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [12896.463381] Key type .llcrypt unregistered [12896.465513] Key type ._llcrypt unregistered [12931.828663] Key type ._llcrypt registered [12931.832795] Key type .llcrypt registered [12931.985593] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [12932.412980] Key type .llcrypt unregistered [12932.418088] Key type ._llcrypt unregistered [13003.195669] Key type ._llcrypt registered [13003.197768] Key type .llcrypt registered [13003.404650] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [13003.781408] Key type .llcrypt unregistered [13003.791570] Key type ._llcrypt unregistered [13035.654444] Key type ._llcrypt registered [13035.656617] Key type .llcrypt registered [13035.749620] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [13035.982897] Key type .llcrypt unregistered [13035.984417] Key type ._llcrypt unregistered [13065.463682] Key type ._llcrypt registered [13065.468811] Key type .llcrypt registered [13065.575990] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [13065.846478] Key type .llcrypt unregistered [13065.848607] Key type ._llcrypt unregistered [13112.345854] Key type ._llcrypt registered [13112.350130] Key type .llcrypt registered [13112.443685] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [13127.172258] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [13128.010334] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13128.034413] alg: No test for adler32 (adler32-zlib) [13129.112172] Lustre: Lustre: Build Version: 2.17.53_79_gff07dcf [13129.314727] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [13130.959131] Key type lgssc registered [13131.890077] Lustre: Echo OBD driver; http://www.lustre.org/ [13162.277290] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13162.304328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13163.636689] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13163.669699] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13163.783852] Lustre: tst250fs-MDT0000: new disk, initializing [13163.903553] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13163.945250] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13167.269236] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [13175.592347] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13175.696124] Lustre: 357834:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13175.775372] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13175.791074] Lustre: Skipped 1 previous similar message [13175.878523] Lustre: tst250fs-MDT0001: new disk, initializing [13175.983356] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13176.026735] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13176.039929] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13179.440286] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [13183.479205] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13188.646165] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13188.828618] Lustre: tst250fs-OST0000: new disk, initializing [13188.833897] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13188.840550] Lustre: 359430:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13188.919931] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13194.315444] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [13194.813438] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13194.822929] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13194.894341] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13203.701758] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13203.879791] Lustre: tst250fs-OST0001: new disk, initializing [13203.883589] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13203.889480] Lustre: 360284:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13203.970782] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13209.192887] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13209.202994] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13209.311715] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [13209.331984] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13214.434737] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13216.600984] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [13216.603857] Lustre: Skipped 2 previous similar messages [13235.174265] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13235.191067] Lustre: Skipped 1 previous similar message [13235.194343] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13235.201528] Lustre: Skipped 2 previous similar messages [13240.258165] Lustre: server umount tst250fs-MDT0000 complete [13243.566932] LustreError: 360283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781250123 with bad export cookie 5827677455752351030 [13243.573534] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13243.580094] LustreError: 360283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13243.843627] Lustre: server umount tst250fs-MDT0001 complete [13257.772930] Lustre: server umount tst250fs-OST0000 complete [13260.767180] Lustre: 356059:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781250125/real 1781250125] req@ffff8ff489a643c0 x1867776009500160/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781250141 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13260.793294] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13260.807243] Lustre: Skipped 2 previous similar messages [13261.157612] Lustre: server umount tst250fs-OST0001 complete [13276.792852] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [13283.251583] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [13310.451734] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 03:43:10 (1781250190) [13311.905477] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13325.415660] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [13333.648249] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [13359.451577] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13095 sec ========= 03:43:59 (1781250239) [13360.580626] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 03:44:00 (1781250240) === [13363.293705] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 03:44:03 (1781250243) === [13396.938772] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [13398.930975] Key type lgssc unregistered [13399.144759] LNet: 369320:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13399.155091] LNetError: 369320:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13399.184496] LNet: Removed LNI 192.168.201.116@tcp [13399.790183] Key type .llcrypt unregistered [13399.792214] Key type ._llcrypt unregistered