[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 477522184 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.988 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002380] x2apic enabled [ 0.003013] Switched APIC routing to physical x2apic. [ 0.004016] kvm-guest: setup PV IPIs [ 0.006887] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22982c12b6d, max_idle_ns: 440795281273 ns [ 0.007029] Calibrating delay loop (skipped) preset value.. 4799.97 BogoMIPS (lpj=2399988) [ 0.009023] pid_max: default: 32768 minimum: 301 [ 0.010153] LSM: Security Framework initializing [ 0.011070] Yama: becoming mindful. [ 0.012041] SELinux: Initializing. [ 0.014071] *** VALIDATE selinux *** [ 0.022786] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027427] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028168] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030018] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031119] *** VALIDATE tmpfs *** [ 0.034212] *** VALIDATE proc *** [ 0.035251] *** VALIDATE cgroup *** [ 0.036011] *** VALIDATE cgroup2 *** [ 0.038121] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039152] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041030] Spectre V2 : User space: Vulnerable [ 0.042007] Speculative Store Bypass: Vulnerable [ 0.045437] debug: unmapping init [mem 0xffffffffb4c59000-0xffffffffb4c60fff] [ 0.048198] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.049753] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.050027] ... version: 2 [ 0.051009] ... bit width: 48 [ 0.051817] ... generic registers: 4 [ 0.052011] ... value mask: 0000ffffffffffff [ 0.053016] ... max period: 00007fffffffffff [ 0.054014] ... fixed-purpose events: 3 [ 0.055013] ... event mask: 000000070000000f [ 0.056330] rcu: Hierarchical SRCU implementation. [ 0.058507] smp: Bringing up secondary CPUs ... [ 0.059620] x86: Booting SMP configuration: [ 0.060029] .... node #0, CPUs: #1 #2 #3 [ 0.064325] smp: Brought up 1 node, 4 CPUs [ 0.066018] smpboot: Max logical packages: 1 [ 0.067018] smpboot: Total of 4 processors activated (19199.90 BogoMIPS) [ 0.147764] node 0 deferred pages initialised in 76ms [ 0.150128] devtmpfs: initialized [ 0.151346] x86/mm: Memory block size: 128MB [ 0.155038] gcov: version magic: 0x41383552 [ 0.158393] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.159082] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.160222] pinctrl core: initialized pinctrl subsystem [ 0.162128] [ 0.162716] ************************************************************* [ 0.164013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.166016] ** ** [ 0.169015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.171012] ** ** [ 0.174016] ** This means that this kernel is built to expose internal ** [ 0.177015] ** IOMMU data structures, which may compromise security on ** [ 0.179013] ** your system. ** [ 0.181013] ** ** [ 0.183013] ** If you see this message and you are not debugging the ** [ 0.185010] ** kernel, report this immediately to your vendor! ** [ 0.187011] ** ** [ 0.189014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.191018] ************************************************************* [ 0.194736] NET: Registered protocol family 16 [ 0.196445] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.199071] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.202113] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.207057] cpuidle: using governor menu [ 0.208689] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.211712] PCI: Using configuration type 1 for base access [ 0.214157] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.224188] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.225065] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.227103] cryptd: max_cpu_qlen set to 1000 [ 0.229197] ACPI: Added _OSI(Module Device) [ 0.230019] ACPI: Added _OSI(Processor Device) [ 0.232018] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.234019] ACPI: Added _OSI(Processor Aggregator Device) [ 0.238510] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.247645] ACPI: Interpreter enabled [ 0.249069] ACPI: PM: (supports S0 S3 S4 S5) [ 0.251020] ACPI: Using IOAPIC for interrupt routing [ 0.253208] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.257461] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.267794] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.271051] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.273021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.277141] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.283735] acpiphp: Slot [2] registered [ 0.285259] acpiphp: Slot [5] registered [ 0.287176] acpiphp: Slot [6] registered [ 0.288186] acpiphp: Slot [7] registered [ 0.290125] acpiphp: Slot [8] registered [ 0.291129] acpiphp: Slot [9] registered [ 0.292103] acpiphp: Slot [10] registered [ 0.293102] acpiphp: Slot [3] registered [ 0.294082] acpiphp: Slot [4] registered [ 0.295107] acpiphp: Slot [11] registered [ 0.296104] acpiphp: Slot [12] registered [ 0.298083] acpiphp: Slot [13] registered [ 0.299125] acpiphp: Slot [14] registered [ 0.301132] acpiphp: Slot [15] registered [ 0.302092] acpiphp: Slot [16] registered [ 0.303028] acpiphp: Slot [17] registered [ 0.304069] acpiphp: Slot [18] registered [ 0.306110] acpiphp: Slot [19] registered [ 0.307138] acpiphp: Slot [20] registered [ 0.309124] acpiphp: Slot [21] registered [ 0.311143] acpiphp: Slot [22] registered [ 0.312113] acpiphp: Slot [23] registered [ 0.314131] acpiphp: Slot [24] registered [ 0.315109] acpiphp: Slot [25] registered [ 0.317115] acpiphp: Slot [26] registered [ 0.318163] acpiphp: Slot [27] registered [ 0.320107] acpiphp: Slot [28] registered [ 0.321122] acpiphp: Slot [29] registered [ 0.323209] acpiphp: Slot [30] registered [ 0.324101] acpiphp: Slot [31] registered [ 0.325112] PCI host bridge to bus 0000:00 [ 0.327025] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.329028] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.332027] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.335028] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.337022] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.340036] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.341351] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.344339] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.347800] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.359935] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.366070] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.369034] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.373047] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.376027] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.378347] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.380768] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.384056] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.387945] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.393014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.405021] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.408839] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.414325] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.425019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.433020] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.450018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.458848] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.466012] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.474013] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.492076] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.502714] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.508017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.514020] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.531018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.544256] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.551026] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.557074] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.576027] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.585104] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.591030] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.599022] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.614019] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.623590] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.636018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.642017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.664017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.680640] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.683597] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.686570] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.689505] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.692339] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.698036] iommu: Default domain type: Passthrough [ 0.700522] SCSI subsystem initialized [ 0.702220] ACPI: bus type USB registered [ 0.703115] usbcore: registered new interface driver usbfs [ 0.706134] usbcore: registered new interface driver hub [ 0.708105] usbcore: registered new device driver usb [ 0.710219] pps_core: LinuxPPS API ver. 1 registered [ 0.712024] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.717137] PTP clock support registered [ 0.720053] EDAC MC: Ver: 3.0.0 [ 0.723112] PCI: Using ACPI for IRQ routing [ 0.724640] NetLabel: Initializing [ 0.726016] NetLabel: domain hash size = 128 [ 0.728012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.730095] NetLabel: unlabeled traffic allowed by default [ 0.732093] vgaarb: loaded [ 0.733385] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.735028] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.741698] clocksource: Switched to clocksource kvm-clock [ 0.856073] VFS: Disk quotas dquot_6.6.0 [ 0.857698] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.859623] *** VALIDATE ramfs *** [ 0.860462] *** VALIDATE hugetlbfs *** [ 0.862340] pnp: PnP ACPI init [ 0.864600] pnp: PnP ACPI: found 6 devices [ 0.883448] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.887461] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.890075] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.892452] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.895345] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.898165] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.901359] NET: Registered protocol family 2 [ 0.904453] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.910523] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.914710] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.921173] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.925391] TCP: Hash tables configured (established 65536 bind 65536) [ 0.928848] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.932591] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.936324] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.939691] NET: Registered protocol family 1 [ 0.943295] RPC: Registered named UNIX socket transport module. [ 0.945675] RPC: Registered udp transport module. [ 0.947216] RPC: Registered tcp transport module. [ 0.950190] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.953178] NET: Registered protocol family 44 [ 0.955351] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.958091] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.960895] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.963842] PCI: CLS 0 bytes, default 64 [ 0.965796] Unpacking initramfs... [ 2.388392] debug: unmapping init [mem 0xffff97e97cc54000-0xffff97e97ffbffff] [ 2.392156] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.394402] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.397813] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22982c12b6d, max_idle_ns: 440795281273 ns [ 2.882487] Initialise system trusted keyrings [ 2.884480] Key type blacklist registered [ 2.886759] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.903287] zbud: loaded [ 2.906957] *** VALIDATE nfs *** [ 2.908192] *** VALIDATE nfs4 *** [ 2.909789] pstore: using deflate compression [ 2.913202] Platform Keyring initialized [ 3.040470] NET: Registered protocol family 38 [ 3.042659] Key type asymmetric registered [ 3.044634] Asymmetric key parser 'x509' registered [ 3.047001] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.050900] io scheduler mq-deadline registered [ 3.052844] io scheduler kyber registered [ 3.054700] io scheduler bfq registered [ 3.056737] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.059684] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.062996] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.065866] ACPI: Power Button [PWRF] [ 3.072250] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.079333] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.093625] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.108159] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.130599] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.158692] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.187798] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.193661] Non-volatile memory driver v1.3 [ 3.195601] Linux agpgart interface v0.103 [ 3.230808] virtio_blk virtio1: [vda] 136032 512-byte logical blocks (69.6 MB/66.4 MiB) [ 3.235877] vda: detected capacity change from 0 to 69648384 [ 3.252425] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.255394] vdb: detected capacity change from 0 to 1073741824 [ 3.270071] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.273927] vdc: detected capacity change from 0 to 2621440000 [ 3.291826] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.295034] vdd: detected capacity change from 0 to 2621440000 [ 3.315527] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.318050] vde: detected capacity change from 0 to 4294967296 [ 3.335382] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.338797] vdf: detected capacity change from 0 to 4294967296 [ 3.347436] libphy: Fixed MDIO Bus: probed [ 3.355318] usbcore: registered new interface driver usbserial_generic [ 3.357843] usbserial: USB Serial support registered for generic [ 3.360405] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.367272] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.369199] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.372090] mousedev: PS/2 mouse device common for all mice [ 3.375674] rtc_cmos 00:05: RTC can wake from S4 [ 3.380132] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.380408] rtc_cmos 00:05: registered as rtc0 [ 3.384294] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.385966] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.387601] intel_pstate: CPU model not supported [ 3.392381] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.396190] hid: raw HID events driver (C) Jiri Kosina [ 3.398922] usbcore: registered new interface driver usbhid [ 3.400981] usbhid: USB HID core driver [ 3.402462] drop_monitor: Initializing network drop monitor service [ 3.404632] Initializing XFRM netlink socket [ 3.405849] NET: Registered protocol family 10 [ 3.408951] Segment Routing with IPv6 [ 3.410654] NET: Registered protocol family 17 [ 3.412652] mpls_gso: MPLS GSO support [ 3.418592] RAS: Correctable Errors collector initialized. [ 3.420823] AVX version of gcm_enc/dec engaged. [ 3.422339] AES CTR mode by8 optimization enabled [ 3.517191] sched_clock: Marking stable (3517150061, 0)->(4467069143, -949919082) [ 3.521195] registered taskstats version 1 [ 3.523911] Loading compiled-in X.509 certificates [ 3.526727] zswap: loaded using pool lzo/zbud [ 3.555271] Key type big_key registered [ 3.568948] Key type encrypted registered [ 3.580925] ima: No TPM chip found, activating TPM-bypass! [ 3.583196] ima: Allocated hash algorithm: sha1 [ 3.585075] ima: No architecture policies found [ 3.586839] evm: Initialising EVM extended attributes: [ 3.589560] evm: security.selinux [ 3.591140] evm: security.ima [ 3.592304] evm: security.capability [ 3.593633] evm: HMAC attrs: 0x1 [ 3.598094] rtc_cmos 00:05: setting system clock to 2026-06-02 07:02:34 UTC (1780383754) [ 3.604426] debug: unmapping init [mem 0xffffffffb5c03000-0xffffffffb5dfffff] [ 3.607568] debug: unmapping init [mem 0xffffffffb4982000-0xffffffffb4c58fff] [ 3.616107] Write protecting the kernel read-only data: 28672k [ 3.619869] debug: unmapping init [mem 0xffffffffb3003000-0xffffffffb31fffff] [ 3.623188] debug: unmapping init [mem 0xffffffffb3914000-0xffffffffb39fffff] [ 3.682084] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.693066] systemd[1]: Detected virtualization kvm. [ 3.695131] systemd[1]: Detected architecture x86-64. [ 3.698188] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.727766] systemd[1]: No hostname configured. [ 3.729564] systemd[1]: Set hostname to . [ 3.732220] random: systemd: uninitialized urandom read (16 bytes read) [ 3.735324] systemd[1]: Initializing machine ID from random generator. [ 3.870200] random: systemd: uninitialized urandom read (16 bytes read) [ 3.873659] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.879265] random: systemd: uninitialized urandom read (16 bytes read) [ 3.881634] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.886418] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Journal Service... [ OK ] Reached target Slices. [ OK ] Reached target Local Encrypted Volumes. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.546277] device-mapper: uevent: version 1.0.3 [ 4.548745] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.254360] virtio_net virtio0 ens2: renamed from eth0 [ 5.279505] random: fast init done [ 5.334236] scsi host0: ata_piix [ 5.367953] scsi host1: ata_piix [ 5.371958] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.374910] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.042983] dracut-initqueue[584]: RTNETLINK answers: File exists [ 10.100779] random: crng init done [ 10.102533] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.602262] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.902323] printk: systemd: 24 output lines suppressed due to ratelimiting [ 12.174342] SELinux: Disabled at runtime. [ 12.234565] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.242887] systemd[1]: Detected virtualization kvm. [ 12.244415] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.824422] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.830467] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.837988] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.842983] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.846879] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.856050] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.865798] systemd[1]: Mounting POSIX Message Queue File System... Mounting POSIX Message Queue File System... [ OK ] Created slice User and Session Slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on udev Kernel Socket. Mounting Kernel Debug File System... [ OK ] Listening on udev Control Socket. Starting Remount Root and Kernel File Systems... [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-getty.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target rpc_pipefs.target. Starting Create list of required st…ce nodes for the current kernel... Mounting Huge Pages File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for det[ 13.005056] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS ails. Starting Apply Kernel Variables... Starting udev Coldplug all Devices... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Slices. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices.[ 13.405778] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.797972] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.262890] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.382128] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.394708] EDAC sbridge: Ver: 1.1.2 [ 16.344815] Key type dns_resolver registered [ 16.668252] NFS: Registering the id_resolver key type [ 16.670785] Key type id_resolver registered [ 16.672813] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg408-server login: [ 45.416523] libcfs: loading out-of-tree module taints kernel. [ 45.438533] Key type ._llcrypt registered [ 45.440546] Key type .llcrypt registered [ 45.495571] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [ 56.077635] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 57.265108] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 57.282366] alg: No test for adler32 (adler32-zlib) [ 58.576570] Lustre: Lustre: Build Version: 2.17.53_57_g7d1c020 [ 59.449270] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [ 61.192200] Key type lgssc registered [ 62.805542] Lustre: Echo OBD driver; http://www.lustre.org/ [ 80.402393] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 106.177030] hrtimer: interrupt took 4021965 ns [ 131.684704] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 148.717564] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 148.820440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 150.441268] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 150.564790] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 150.932772] Lustre: lustre-MDT0000: new disk, initializing [ 151.169594] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 151.226340] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 159.392347] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 177.733227] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 177.903628] Lustre: 6481:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 177.943660] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 177.952621] Lustre: Skipped 1 previous similar message [ 178.158865] Lustre: lustre-MDT0001: new disk, initializing [ 178.261800] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 178.297721] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 178.316250] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 183.232840] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 187.960425] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 200.379517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 200.900380] Lustre: lustre-OST0000: new disk, initializing [ 200.906933] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 200.915810] Lustre: 8385:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 201.037131] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 208.416225] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 208.433589] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 208.561654] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 209.337323] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 225.431959] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 225.600593] Lustre: lustre-OST0001: new disk, initializing [ 225.603729] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 225.608923] Lustre: 9443:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 225.698210] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 232.590091] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 233.521823] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 233.544238] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 233.628328] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 245.325705] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 257.791165] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 266.631741] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing check_logdir /tmp/testlogs/ [ 274.639429] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing yml_node [ 279.975221] Lustre: DEBUG MARKER: Client: 2.17.53.57 [ 283.891617] Lustre: DEBUG MARKER: MDS: 2.17.53.57 [ 287.504258] Lustre: DEBUG MARKER: OSS: 2.17.53.57 [ 290.072741] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Jun 2 03:07:18 EDT 2026 [ 310.373806] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 311.955222] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 330.723286] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 330.728990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 330.737501] Lustre: Skipped 3 previous similar messages [ 330.743580] Lustre: Skipped 2 previous similar messages [ 334.538059] Lustre: server umount lustre-MDT0000 complete [ 338.097464] LustreError: 6473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780384088 with bad export cookie 18288234222647819066 [ 338.101654] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 338.106976] LustreError: 6473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 340.977069] LustreError: 6487:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 340.978212] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 340.979216] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 340.979223] Lustre: Skipped 1 previous similar message [ 340.995772] LustreError: 6487:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 341.004107] Lustre: Skipped 1 previous similar message [ 343.975512] LustreError: 6489:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 343.979285] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 344.013879] LustreError: 6489:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 344.049315] Lustre: Skipped 2 previous similar messages [ 344.481578] Lustre: server umount lustre-MDT0001 complete [ 348.431585] Lustre: server umount lustre-OST0000 complete [ 352.640380] Lustre: server umount lustre-OST0001 complete [ 360.048749] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [ 371.355342] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 409.167910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 409.562538] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 409.631612] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 409.786075] Lustre: lustre-MDT0000: new disk, initializing [ 409.905198] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 409.925552] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 414.521997] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 426.239105] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 426.359665] Lustre: 14964:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 426.437761] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 426.442439] Lustre: Skipped 1 previous similar message [ 426.509830] Lustre: lustre-MDT0001: new disk, initializing [ 426.635689] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 426.687589] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 426.714629] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 431.263594] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 439.699993] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 443.233642] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 449.953500] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 450.355955] Lustre: lustre-OST0000: new disk, initializing [ 450.366114] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 450.376801] Lustre: 16705:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 450.508465] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 451.814384] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 451.830079] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 451.982153] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 456.549279] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 464.162141] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 468.739968] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 468.949539] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 473.205504] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 473.455894] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 477.665483] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 477.671511] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 477.683256] LustreError: Skipped 1 previous similar message [ 477.684110] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 477.712406] Lustre: Skipped 1 previous similar message [ 481.627517] Lustre: server umount lustre-OST0000 complete [ 487.911325] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 487.914362] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 487.917615] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 487.917623] Lustre: Skipped 1 previous similar message [ 487.934401] Lustre: Skipped 1 previous similar message [ 497.123911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 497.130699] Lustre: Skipped 4 previous similar messages [ 500.192228] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 500.672549] Lustre: server umount lustre-MDT0000 complete [ 502.247283] LustreError: 15756:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 502.272925] LustreError: 15756:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 504.915266] LustreError: 14955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780384255 with bad export cookie 18288234222647820599 [ 504.926046] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 504.931023] LustreError: 14955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 505.345158] Lustre: server umount lustre-MDT0001 complete [ 512.077822] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 03:11:01 (1780384261) [ 530.611756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 530.940058] LustreError: 19434:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 531.045663] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 535.344903] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 536.044066] LustreError: 19435:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 536.067950] LustreError: 19435:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 544.419624] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 550.081961] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 555.676858] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 559.291578] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 561.685253] Lustre: Failing over lustre-MDT0000 [ 561.925128] Lustre: server umount lustre-MDT0000 complete [ 565.219232] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 565.223308] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 565.240523] LustreError: 20145:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 565.259288] LustreError: 20145:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 582.112100] Lustre: 3632:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780384316/real 1780384316] req@ffff97e8c6c88000 x1866867734126208/t0(0) o400->MGC192.168.204.108@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1780384332 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 582.131363] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 582.166134] LustreError: 19434:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 582.190548] LustreError: 19434:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 584.840876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 592.353885] LustreError: 3629:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff97e8c6c23800 x1866867734128896/t0(0) o250->MGC192.168.204.108@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 597.749909] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 597.987452] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 598.007624] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 598.028695] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 605.029634] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 612.321841] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 612.332184] Lustre: Skipped 1 previous similar message [ 612.335737] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 612.339783] Lustre: Skipped 1 previous similar message [ 615.284337] Lustre: server umount lustre-MDT0000 complete [ 619.002185] LustreError: 20153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780384369 with bad export cookie 18288234222647822013 [ 619.006581] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 619.010029] LustreError: 20153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 619.283950] Lustre: server umount lustre-MDT0001 complete [ 627.894206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 628.315862] LustreError: 22433:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 628.333134] LustreError: 22433:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 628.424688] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 628.431787] Lustre: Skipped 2 previous similar messages [ 633.263730] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 642.220613] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 646.830436] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 652.727428] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 656.230147] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 658.471980] Lustre: Failing over lustre-MDT0000 [ 658.493739] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 658.495242] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 658.508761] Lustre: Skipped 2 previous similar messages [ 658.527112] Lustre: Skipped 3 previous similar messages [ 658.681746] Lustre: server umount lustre-MDT0000 complete [ 678.002893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 678.250525] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 678.734477] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 683.198677] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 684.004297] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 684.023972] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 684.040104] Lustre: Skipped 1 previous similar message [ 684.068411] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 690.341737] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 694.246564] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 694.253658] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 700.094365] Lustre: server umount lustre-MDT0000 complete [ 703.314135] LustreError: 24754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780384454 with bad export cookie 18288234222647822951 [ 703.317518] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 703.323803] LustreError: 24754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 703.589374] Lustre: server umount lustre-MDT0001 complete [ 711.700801] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 03:14:20 (1780384460) [ 725.392144] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [ 733.733938] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 765.882661] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 775.514490] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 775.735771] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 775.767068] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 775.837924] Lustre: lustre-MDT0000: new disk, initializing [ 775.933752] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 775.940870] Lustre: Skipped 2 previous similar messages [ 775.963539] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 779.695831] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 789.237568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 789.367294] Lustre: 28412:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 789.606237] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 793.513433] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 800.484728] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 804.215981] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 809.832784] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 810.170329] Lustre: 30153:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 811.644048] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 811.700893] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 811.769526] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 811.788335] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 815.707719] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 822.327711] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 827.289694] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 852.965134] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 852.972133] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 852.984177] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 852.987941] Lustre: Skipped 2 previous similar messages [ 853.011690] Lustre: Skipped 4 previous similar messages [ 854.537622] Lustre: server umount lustre-OST0000 complete [ 864.243634] Lustre: server umount lustre-MDT0000 complete [ 867.718931] LustreError: 29190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780384618 with bad export cookie 18288234222647823357 [ 867.722708] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 867.739669] LustreError: 29190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 868.114800] Lustre: server umount lustre-MDT0001 complete [ 876.843378] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 879.102868] Key type lgssc unregistered [ 879.342656] LNet: 32001:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 879.347604] LNetError: 32001:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 880.358720] LNet: Removed LNI 192.168.204.108@tcp [ 881.082150] Key type .llcrypt unregistered [ 881.087883] Key type ._llcrypt unregistered [ 900.714649] Key type ._llcrypt registered [ 900.718451] Key type .llcrypt registered [ 900.809692] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 901.326323] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 901.348023] alg: No test for adler32 (adler32-zlib) [ 902.324819] Lustre: Lustre: Build Version: 2.17.53_57_g7d1c020 [ 902.486709] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [ 904.128206] Key type lgssc registered [ 905.180429] Lustre: Echo OBD driver; http://www.lustre.org/ [ 912.123868] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 912.160140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 913.598569] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 917.001243] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 922.833318] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 923.074834] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 926.357935] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 930.564106] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 933.140110] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 937.861025] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 938.110449] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 943.143648] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 947.317874] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 947.743959] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 958.891278] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 03:18:28 (1780384708) [ 960.527430] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 961.961701] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 967.492863] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 03:18:36 (1780384716) [ 974.432765] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 03:18:43 (1780384723) [ 975.336754] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 982.272316] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 982.278764] Lustre: Skipped 2 previous similar messages [ 983.861541] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 03:18:53 (1780384733) [ 987.112715] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 987.120600] Lustre: Skipped 3 previous similar messages [ 1001.959419] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1001.968419] Lustre: Skipped 6 previous similar messages [ 1004.961230] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 03:19:13 (1780384753) [ 1007.680697] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1010.491585] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1017.648825] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 03:19:26 (1780384766) [ 1019.404590] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1019.413574] Lustre: Skipped 3 previous similar messages [ 1029.575222] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 03:19:38 (1780384778) [ 1041.890394] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1041.898488] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1041.906782] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1044.963285] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1044.963744] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1044.969335] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1046.819440] Lustre: server umount lustre-OST0000 complete [ 1050.085220] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1050.101707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1050.107417] Lustre: Skipped 2 previous similar messages [ 1055.203389] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1055.208824] Lustre: Skipped 1 previous similar message [ 1056.197400] Lustre: server umount lustre-MDT0000 complete [ 1059.205684] LustreError: 33361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780384810 with bad export cookie 10145853816118541330 [ 1059.208736] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1059.213909] LustreError: 33361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1059.459802] Lustre: server umount lustre-MDT0001 complete [ 1067.060713] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 1068.810707] Key type lgssc unregistered [ 1069.018821] LNet: 40875:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1069.025941] LNetError: 40875:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1069.041259] LNet: Removed LNI 192.168.204.108@tcp [ 1069.569588] Key type .llcrypt unregistered [ 1069.573894] Key type ._llcrypt unregistered [ 1082.486757] Key type ._llcrypt registered [ 1082.490595] Key type .llcrypt registered [ 1082.563342] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [ 1094.111293] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 1094.725561] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1094.771903] alg: No test for adler32 (adler32-zlib) [ 1095.804521] Lustre: Lustre: Build Version: 2.17.53_57_g7d1c020 [ 1096.050768] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [ 1097.736191] Key type lgssc registered [ 1098.806152] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1127.021530] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1127.097107] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1128.294773] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1128.330128] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1128.399708] Lustre: lustre-MDT0000: new disk, initializing [ 1128.516562] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1128.560957] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1132.158233] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 1139.581816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1139.661884] Lustre: 43971:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1139.712900] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1139.719178] Lustre: Skipped 1 previous similar message [ 1139.781273] Lustre: lustre-MDT0001: new disk, initializing [ 1139.859383] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1139.885966] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1139.891705] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1142.540209] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 1147.829195] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1150.064348] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1154.030613] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1154.229379] Lustre: lustre-OST0000: new disk, initializing [ 1154.232410] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1154.237419] Lustre: 45710:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1154.285867] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1157.816281] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 1163.314513] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1164.304258] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1164.312105] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1164.363682] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1165.875242] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1165.995689] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1168.849802] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1168.985600] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1174.501332] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1174.505161] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1174.514168] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1174.526110] Lustre: Skipped 1 previous similar message [ 1174.533157] LustreError: Skipped 1 previous similar message [ 1176.376403] Lustre: server umount lustre-OST0000 complete [ 1179.619567] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1179.623426] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1179.636087] Lustre: Skipped 1 previous similar message [ 1179.650987] Lustre: Skipped 2 previous similar messages [ 1184.739250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1184.744290] Lustre: Skipped 1 previous similar message [ 1185.174716] Lustre: server umount lustre-MDT0000 complete [ 1187.719429] LustreError: 43964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780384938 with bad export cookie 9259926866927142166 [ 1187.722226] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1187.724946] LustreError: 43964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1187.990218] Lustre: server umount lustre-MDT0001 complete [ 1195.616313] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 03:22:24 (1780384944) [ 1201.983993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1202.369870] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1206.103210] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 1207.778972] LustreError: 48022:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1207.794818] LustreError: 48022:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1212.904636] LustreError: 48023:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1213.094913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1213.334688] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1217.306783] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 1222.446551] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1226.030406] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1232.191934] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1232.622764] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1237.844164] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 1243.149690] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1251.732214] LustreError: 50211:0:(llog.c:444:llog_init_handle()) llog type is not specified! [ 1252.474603] Lustre: Modifying parameter general.timeout=1 in log params [ 1253.793315] Lustre: Modifying parameter general.timeout=3 in log params [ 1253.797495] Lustre: Skipped 1 previous similar message [ 1255.849587] Lustre: Modifying parameter general.timeout=6 in log params [ 1255.854105] Lustre: Skipped 2 previous similar messages [ 1260.363299] Lustre: Modifying parameter general.timeout=12 in log params [ 1260.367459] Lustre: Skipped 5 previous similar messages [ 1268.528244] Lustre: Modifying parameter general.timeout=24 in log params [ 1268.532263] Lustre: Skipped 11 previous similar messages [ 1284.751785] Lustre: Modifying parameter general.timeout=48 in log params [ 1284.755895] Lustre: Skipped 23 previous similar messages [ 1317.416107] Lustre: Modifying parameter general.timeout=97 in log params [ 1317.420474] Lustre: Skipped 48 previous similar messages [ 1345.266114] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 03:24:54 (1780385094) [ 1347.642293] Lustre: 57140:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1349.091499] Lustre: 57239:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1352.146844] Lustre: 57455:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1352.167033] Lustre: 57455:0:(mgs_llog.c:1437:mgs_modify_param()) Skipped 1 previous similar message [ 1373.912727] Lustre: 59001:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1373.918575] Lustre: 59001:0:(mgs_llog.c:1437:mgs_modify_param()) Skipped 2 previous similar messages [ 1373.934456] LustreError: 59001:0:(mgs_handler.c:1202:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1376.752199] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 03:25:26 (1780385126) [ 1384.418109] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1384.420478] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1384.426206] Lustre: Skipped 2 previous similar messages [ 1384.431652] Lustre: Skipped 2 previous similar messages [ 1389.539921] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1389.548403] Lustre: Skipped 2 previous similar messages [ 1390.095915] Lustre: server umount lustre-MDT0000 complete [ 1392.788647] LustreError: 52290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780385143 with bad export cookie 9259926866927143048 [ 1392.789607] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1392.796384] LustreError: 52290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1392.979707] Lustre: server umount lustre-MDT0001 complete [ 1406.248301] Lustre: server umount lustre-OST0000 complete [ 1425.684223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1425.893412] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1425.926245] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1425.936358] Lustre: Skipped 55 previous similar messages [ 1426.080163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1428.991340] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 1436.024973] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1436.106649] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1436.124530] Lustre: 61528:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1439.372548] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 1442.867230] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1445.607118] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1451.006591] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1451.186457] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1455.918749] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 1460.040502] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1461.527539] Lustre: 63757:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1467.873120] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1467.873830] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1467.880958] LustreError: Skipped 1 previous similar message [ 1467.897569] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1472.285205] Lustre: server umount lustre-OST0000 complete [ 1477.602535] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1477.608198] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1477.620872] Lustre: Skipped 1 previous similar message [ 1487.334710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1487.339411] Lustre: Skipped 7 previous similar messages [ 1489.377148] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1489.513679] Lustre: server umount lustre-MDT0000 complete [ 1491.723775] LustreError: 61567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780385242 with bad export cookie 9259926866927145855 [ 1491.728382] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1491.732985] LustreError: 61567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1491.912923] Lustre: server umount lustre-MDT0001 complete [ 1498.240204] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 1500.151049] Key type lgssc unregistered [ 1500.383803] LNet: 64951:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1500.393815] LNetError: 64951:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1500.405061] LNet: Removed LNI 192.168.204.108@tcp [ 1501.023332] Key type .llcrypt unregistered [ 1501.025784] Key type ._llcrypt unregistered [ 1517.755626] Key type ._llcrypt registered [ 1517.758761] Key type .llcrypt registered [ 1517.846232] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 1518.518257] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1518.535174] alg: No test for adler32 (adler32-zlib) [ 1519.498930] Lustre: Lustre: Build Version: 2.17.53_57_g7d1c020 [ 1519.654128] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [ 1521.272231] Key type lgssc registered [ 1522.054489] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1526.938880] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1526.954587] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1528.326465] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1530.848371] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 1536.338405] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1536.725204] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1539.115925] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 1542.448992] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1544.465886] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1548.753542] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1549.147532] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1553.373283] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 1556.890344] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1566.849485] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1572.323131] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1572.332824] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1572.343888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1576.930609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1576.930828] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1576.936820] Lustre: Skipped 1 previous similar message [ 1576.948117] Lustre: Skipped 1 previous similar message [ 1578.338414] Lustre: server umount lustre-MDT0000 complete [ 1580.607274] LustreError: 66305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780385331 with bad export cookie 10119989342177915501 [ 1580.613388] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1580.796746] Lustre: server umount lustre-MDT0001 complete [ 1593.540248] Lustre: server umount lustre-OST0000 complete [ 1627.956442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1628.328980] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1628.413474] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1628.776700] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1634.650845] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 1646.667289] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1646.928589] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1647.024618] Lustre: 70859:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1647.607277] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1652.581261] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 1658.500621] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1663.045586] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1669.468241] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1669.680995] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1670.043619] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1674.926802] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 1680.316023] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1681.898344] Lustre: 73092:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1683.677737] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1690.082253] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1690.091561] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1690.104962] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1690.109741] Lustre: Skipped 1 previous similar message [ 1693.669361] Lustre: server umount lustre-OST0000 complete [ 1698.785728] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1698.791494] LustreError: Skipped 1 previous similar message [ 1698.795362] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1698.814431] Lustre: Skipped 1 previous similar message [ 1698.819644] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1698.825434] Lustre: Skipped 1 previous similar message [ 1703.836126] Lustre: server umount lustre-MDT0000 complete [ 1705.956575] LustreError: 70867:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1705.983211] LustreError: 70867:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1707.267789] LustreError: 70851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780385458 with bad export cookie 10119989342177916313 [ 1707.270404] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1707.278318] LustreError: 70851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1707.715377] Lustre: server umount lustre-MDT0001 complete [ 1718.289476] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 1720.625362] Key type lgssc unregistered [ 1720.941387] LNet: 74283:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1720.948320] LNetError: 74283:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1720.967410] LNet: Removed LNI 192.168.204.108@tcp [ 1721.870415] Key type .llcrypt unregistered [ 1721.875647] Key type ._llcrypt unregistered [ 1745.649219] Key type ._llcrypt registered [ 1745.652081] Key type .llcrypt registered [ 1745.750816] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 1746.704240] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1746.810894] alg: No test for adler32 (adler32-zlib) [ 1747.812995] Lustre: Lustre: Build Version: 2.17.53_57_g7d1c020 [ 1748.076991] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [ 1749.769327] Key type lgssc registered [ 1750.987713] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1758.759109] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1758.787421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1760.320177] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1764.602989] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 1770.480685] LustreError: 75678:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1770.497971] LustreError: 75678:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1775.586245] LustreError: 75677:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1776.311755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1777.018639] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1782.600939] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 1790.042545] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1794.289918] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1801.968806] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1803.121560] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1809.908282] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 1815.813482] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1825.504895] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1826.454760] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1828.185106] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1828.194545] Lustre: Skipped 1 previous similar message [ 1830.330558] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1830.334701] Lustre: Skipped 2 previous similar messages [ 1834.451371] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1834.461615] Lustre: Skipped 4 previous similar messages [ 1842.502563] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1842.505899] Lustre: Skipped 8 previous similar messages [ 1858.818357] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1858.826222] Lustre: Skipped 16 previous similar messages [ 1891.760645] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1891.769710] Lustre: Skipped 35 previous similar messages [ 1930.230590] Lustre: 83449:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 1932.857073] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 03:34:41 (1780385681) [ 1934.340476] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 1936.010764] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 03:34:45 (1780385685) [ 1939.444602] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1939.445126] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1939.457578] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1939.479200] Lustre: Skipped 1 previous similar message [ 1939.496040] Lustre: Skipped 1 previous similar message [ 1944.562371] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1944.583125] Lustre: Skipped 1 previous similar message [ 1945.663178] Lustre: server umount lustre-OST0000 complete [ 1949.666151] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1949.671293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1949.676958] Lustre: Skipped 1 previous similar message [ 1949.688091] Lustre: Skipped 1 previous similar message [ 1954.791561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1954.808953] Lustre: Skipped 1 previous similar message [ 1955.503459] Lustre: server umount lustre-MDT0000 complete [ 1959.191031] LustreError: 75657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780385710 with bad export cookie 17967296958945759819 [ 1959.206328] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1959.555514] Lustre: server umount lustre-MDT0001 complete [ 1968.705214] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 1970.927650] Key type lgssc unregistered [ 1971.141869] LNet: 84680:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1971.148594] LNetError: 84680:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1971.163090] LNet: Removed LNI 192.168.204.108@tcp [ 1971.853923] Key type .llcrypt unregistered [ 1971.857208] Key type ._llcrypt unregistered [ 1975.125491] Key type ._llcrypt registered [ 1975.127755] Key type .llcrypt registered [ 1975.213323] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 1979.142740] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules [ 1980.256973] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1980.280850] alg: No test for adler32 (adler32-zlib) [ 1981.524541] Lustre: Lustre: Build Version: 2.17.53_57_g7d1c020 [ 1981.848497] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [ 1983.504609] Key type lgssc registered [ 1984.700666] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1985.634089] LustreError: 85849:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 1999.993445] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 2007.273853] LustreError: 85849:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2007.301440] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2007.327760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2008.761907] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2012.292233] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 2018.282514] LustreError: 86856:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2018.296674] LustreError: 86856:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2020.413450] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 03:36:09 (1780385769) [ 2023.398410] LustreError: 86855:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2024.936947] LustreError: 86887:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2024.955209] LustreError: 86887:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2024.969636] LustreError: 86887:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 16, retries 0, failed: rc = -5 [ 2025.349794] Lustre: server umount lustre-MDT0000 complete [ 2037.539114] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 2040.058288] Key type lgssc unregistered [ 2040.310741] LNet: 88345:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2040.317757] LNetError: 88345:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2041.384900] LNet: Removed LNI 192.168.204.108@tcp [ 2042.111164] Key type .llcrypt unregistered [ 2042.112994] Key type ._llcrypt unregistered [ 2054.969688] Key type ._llcrypt registered [ 2054.974868] Key type .llcrypt registered [ 2055.063249] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 2055.746687] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2055.854685] alg: No test for adler32 (adler32-zlib) [ 2056.915662] Lustre: Lustre: Build Version: 2.17.53_57_g7d1c020 [ 2057.099672] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [ 2058.744253] Key type lgssc registered [ 2059.719506] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2066.342823] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2066.367739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2067.917585] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2071.320251] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 2077.158480] LustreError: 89525:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2077.169777] LustreError: 89525:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2078.017957] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2078.540266] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2082.456442] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 2086.586481] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2089.076179] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2093.715603] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2094.149207] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2098.432264] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 2102.693084] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2148.410100] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 03:38:17 (1780385897) [ 2149.633198] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2150.888636] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 03:38:20 (1780385900) [ 2153.956392] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2153.959381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2153.965395] Lustre: Skipped 2 previous similar messages [ 2153.979702] Lustre: Skipped 2 previous similar messages [ 2159.082796] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2159.088993] Lustre: Skipped 2 previous similar messages [ 2159.609287] Lustre: server umount lustre-MDT0000 complete [ 2162.579606] LustreError: 89506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780385913 with bad export cookie 5731904775657363283 [ 2162.588047] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2162.589926] LustreError: 89506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2162.788460] Lustre: server umount lustre-MDT0001 complete [ 2175.946095] Lustre: server umount lustre-OST0000 complete [ 2183.298668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2183.609120] LustreError: 93484:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2183.727639] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2186.414532] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 2188.771652] LustreError: 93485:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2192.783936] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2193.311281] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2196.456633] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 2199.825703] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2201.860933] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2210.173514] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2210.308986] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2210.315808] LustreError: 93473:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2210.327919] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2210.332174] LustreError: 95341:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -98 [ 2210.336913] LustreError: 95341:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 2210.342550] LustreError: 95341:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2210.384987] Lustre: server umount lustre-OST0000 complete [ 2210.387413] LustreError: 95341:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2216.076753] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2216.204187] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2216.210286] LustreError: 93473:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2216.216550] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2216.221690] LustreError: 95803:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -98 [ 2216.226715] LustreError: 95803:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 2216.231669] LustreError: 95803:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2216.268806] Lustre: server umount lustre-OST0000 complete [ 2216.271640] LustreError: 95803:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2222.186755] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2222.292943] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2222.295393] Lustre: Found index 0 for lustre-OST0000, updating log [ 2222.299875] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2222.350254] Lustre: lustre-OST0000: new disk, initializing [ 2222.356532] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2222.363284] Lustre: 96336:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2222.509575] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2226.058779] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 2228.338563] Lustre: Failing over lustre-OST0000 [ 2228.464166] Lustre: server umount lustre-OST0000 complete [ 2234.339121] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2234.342935] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2234.350554] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2237.922215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2237.926167] Lustre: Skipped 2 previous similar messages [ 2243.042723] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2243.051669] Lustre: Skipped 1 previous similar message [ 2245.088118] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2245.212733] Lustre: server umount lustre-MDT0000 complete [ 2247.532929] LustreError: 93465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780385998 with bad export cookie 5731904775657366020 [ 2247.536914] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2247.540721] LustreError: 93465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2247.718497] Lustre: server umount lustre-MDT0001 complete [ 2252.803383] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 03:40:02 (1780386002) [ 2262.554889] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 2268.903067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2269.206189] LustreError: 98647:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2269.299246] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2271.747416] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 2274.787466] LustreError: 98648:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2277.029722] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2280.061551] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 2281.708923] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2285.492068] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2288.745401] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 2293.315213] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2293.356118] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2293.409912] Lustre: lustre-OST0001: new disk, initializing [ 2293.412775] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2293.416902] Lustre: 100910:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2296.894683] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 2300.449299] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2300.455154] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2300.532480] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2301.664797] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2306.990698] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2310.631911] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2310.633144] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2310.642256] Lustre: Skipped 3 previous similar messages [ 2310.646039] Lustre: Skipped 1 previous similar message [ 2310.651789] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2315.850452] Lustre: server umount lustre-MDT0001 complete [ 2318.754575] LustreError: 99538:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2322.266325] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2322.332636] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2322.336673] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2322.377325] Lustre: 98635:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2322.841968] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2322.846650] Lustre: Skipped 3 previous similar messages [ 2323.563845] LustreError: 102466:0:(obd_mount.c:225:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2323.570575] LustreError: 102466:0:(tgt_mount.c:668:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2323.577897] LustreError: 98639:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2323.627521] LustreError: 102471:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.108@tcp: cfg command failed: rc = -17 [ 2323.637202] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2323.637202] [ 2325.320100] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 2328.045845] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2330.404259] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 03:41:20 (1780386080) [ 2339.457377] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 2350.166919] Lustre: 104463:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2362.235346] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2364.157359] Lustre: 105598:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2430.436075] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2430.438779] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2430.439032] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2430.439532] Lustre: Skipped 5 previous similar messages [ 2430.445290] Lustre: Skipped 3 previous similar messages [ 2436.171142] Lustre: server umount lustre-MDT0000 complete [ 2438.458218] LustreError: 100909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780386189 with bad export cookie 5731904775657366951 [ 2438.462261] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2438.465560] LustreError: 100909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2438.643708] Lustre: server umount lustre-MDT0001 complete [ 2451.115131] Lustre: server umount lustre-OST0000 complete [ 2463.584248] Lustre: server umount lustre-OST0001 complete [ 2466.745615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2471.629190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2475.878738] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2481.321301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2490.820334] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 2496.243339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2496.258098] Lustre: lustre-MDT0000: reset Object Index mappings [ 2496.533617] LustreError: 109529:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2496.551178] LustreError: 109529:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2496.613547] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2498.370467] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 2503.373672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2503.389365] Lustre: lustre-MDT0001: reset Object Index mappings [ 2506.049878] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 2507.535215] Lustre: 110642:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2510.943992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2513.388104] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:200 to 0x280000400:225) [ 2514.172854] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 2518.459563] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2521.798231] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 2521.900135] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:349 to 0x280000401:386) [ 2521.904193] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 2521.905824] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 2525.981178] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2527.634050] Lustre: 112483:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2554.998765] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 03:45:04 (1780386304) [ 2585.570093] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2585.573957] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2585.585034] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2585.587955] Lustre: Skipped 7 previous similar messages [ 2598.369280] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2598.480137] Lustre: server umount lustre-MDT0000 complete [ 2598.883251] LustreError: 110589:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2598.887747] LustreError: 110589:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 2600.120670] LustreError: 109511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780386351 with bad export cookie 5731904775657413711 [ 2600.126719] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2600.130182] LustreError: 109511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2604.004353] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2604.015102] Lustre: Skipped 4 previous similar messages [ 2606.420302] Lustre: server umount lustre-MDT0001 complete [ 2614.285722] Lustre: server umount lustre-OST0000 complete [ 2622.440983] Lustre: server umount lustre-OST0001 complete [ 2625.192449] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [ 2629.671052] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 2646.543081] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2646.658969] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2646.678417] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2646.718459] Lustre: lustre-MDT0000: new disk, initializing [ 2646.757850] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2646.762037] Lustre: Skipped 3 previous similar messages [ 2646.771961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2648.865746] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 2653.168360] Lustre: server umount lustre-MDT0000 complete [ 2658.311637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2658.486615] Lustre: Found index 0 for lustre-MDT0000, updating log [ 2658.505729] Lustre: 116693:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2658.514792] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 2660.954943] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 2663.414141] Lustre: server umount lustre-MDT0000 complete [ 2667.660656] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 03:46:57 (1780386417) [ 2668.400464] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 2669.299209] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 03:46:59 (1780386419) [ 2677.678847] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 2682.993731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2685.560209] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 2690.567608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2690.636689] Lustre: 118706:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2690.677151] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2690.680459] Lustre: Skipped 1 previous similar message [ 2690.741247] Lustre: lustre-MDT0001: new disk, initializing [ 2690.834390] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2690.847186] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2692.934085] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 2695.989095] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 2699.223199] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2699.408326] Lustre: lustre-OST0000: new disk, initializing [ 2699.411834] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2699.415762] Lustre: 120248:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2699.681948] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2699.689194] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2699.753316] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2702.536265] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 2707.876314] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2707.959245] Lustre: lustre-OST0001: new disk, initializing [ 2707.962280] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2707.965879] Lustre: 121101:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2709.864830] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2709.871296] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2709.916827] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2710.750426] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 2716.850461] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2718.736017] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2918.713455] Lustre: 127472:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 2918.717888] Lustre: 127472:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 2918.720710] Lustre: 127472:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 2918.724741] Lustre: 127472:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 2918.728871] Lustre: 127472:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 2918.731764] Lustre: 127472:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3051.747492] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 03:53:21 (1780386801) [ 3064.801867] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3064.805948] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3064.812795] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3064.816028] Lustre: Skipped 12 previous similar messages [ 3069.710465] Lustre: server umount lustre-MDT0000 complete [ 3071.213574] LustreError: 119429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780386822 with bad export cookie 5731904775657474695 [ 3071.215952] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3071.217938] LustreError: 119429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3073.505518] LustreError: 120256:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3073.512061] LustreError: 120256:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 3077.494655] Lustre: server umount lustre-MDT0001 complete [ 3085.452683] Lustre: server umount lustre-OST0000 complete [ 3093.228292] Lustre: server umount lustre-OST0001 complete [ 3095.684347] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [ 3100.004372] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 3115.726240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3115.842158] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3115.859112] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3115.898903] Lustre: lustre-MDT0000: new disk, initializing [ 3115.937401] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3115.940905] Lustre: Skipped 5 previous similar messages [ 3115.950191] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3117.631212] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 3122.641102] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3122.688983] Lustre: 130462:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3122.816403] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3124.429370] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 3128.019829] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3129.449110] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3132.008914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3132.125895] Lustre: 132202:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3133.184844] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3133.188180] Lustre: Skipped 1 previous similar message [ 3133.190563] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3133.230389] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3134.270599] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 3137.868939] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3141.608778] Lustre: lustre-MDD0000: changelog on [ 3142.697661] Lustre: lustre-MDD0001: changelog on [ 3668.890244] Lustre: lustre-MDD0001: changelog off [ 3669.477119] Lustre: 131484:0:(llog_cat.c:977:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 3669.759115] Lustre: lustre-MDD0000: changelog off [ 3671.520763] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3671.523385] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3671.526823] Lustre: Skipped 5 previous similar messages [ 3671.529289] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3671.531602] Lustre: Skipped 8 previous similar messages [ 3677.329580] Lustre: server umount lustre-OST0000 complete [ 3684.737810] Lustre: server umount lustre-MDT0000 complete [ 3686.001098] LustreError: 130453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780387436 with bad export cookie 5731904775658278792 [ 3686.004108] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3686.005058] LustreError: 130453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3686.128138] Lustre: server umount lustre-MDT0001 complete [ 3689.709804] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 3690.773417] Key type lgssc unregistered [ 3690.907379] LNet: 140699:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3690.909705] LNetError: 140699:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3690.916301] LNet: Removed LNI 192.168.204.108@tcp [ 3691.218124] Key type .llcrypt unregistered [ 3691.219846] Key type ._llcrypt unregistered [ 3698.006834] Key type ._llcrypt registered [ 3698.008491] Key type .llcrypt registered [ 3698.049888] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [ 3704.532136] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 3704.818751] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3704.826541] alg: No test for adler32 (adler32-zlib) [ 3705.689836] Lustre: Lustre: Build Version: 2.17.53_57_g7d1c020 [ 3705.773046] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [ 3707.352128] Key type lgssc registered [ 3707.741469] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3719.925648] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3719.933150] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3721.026523] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3721.040748] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3721.078986] Lustre: lustre-MDT0000: new disk, initializing [ 3721.105163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3721.113924] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3722.475552] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 3726.871843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3726.913781] Lustre: 143838:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3726.940167] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3726.943385] Lustre: Skipped 1 previous similar message [ 3726.983471] Lustre: lustre-MDT0001: new disk, initializing [ 3727.020461] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3727.035242] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3727.039551] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3728.478545] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 3731.843446] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3733.138044] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3735.462085] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3735.557801] Lustre: lustre-OST0000: new disk, initializing [ 3735.559518] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3735.561680] Lustre: 145579:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3735.591704] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3736.375114] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3736.377962] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3736.408726] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3737.418533] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 3740.437158] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3742.626785] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_module obdecho/obdecho [ 3743.256345] LustreError: 146452:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 3743.261058] LustreError: 146452:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 3743.860974] LustreError: 146548:0:(echo_client.c:1883:echo_md_handler()) Can not resolve the path : rc = -2 [ 3747.808858] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3747.811611] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3747.816692] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3751.906791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3751.907017] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3751.909981] Lustre: Skipped 1 previous similar message [ 3751.916098] Lustre: Skipped 1 previous similar message [ 3753.558244] Lustre: server umount lustre-MDT0000 complete [ 3754.785032] LustreError: 144617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780387505 with bad export cookie 11812862482553923480 [ 3754.789264] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3754.789680] LustreError: 144617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3754.893114] Lustre: server umount lustre-MDT0001 complete [ 3766.427406] Lustre: server umount lustre-OST0000 complete [ 3773.159470] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [ 3776.494616] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 3787.917901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3788.006579] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3788.016445] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3788.048984] Lustre: lustre-MDT0000: new disk, initializing [ 3788.068757] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3788.074516] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3789.217792] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 3793.373846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3793.414235] Lustre: 150218:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3793.437290] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3793.441015] Lustre: Skipped 1 previous similar message [ 3793.489603] Lustre: lustre-MDT0001: new disk, initializing [ 3793.527047] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3793.540679] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3793.544150] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3794.899546] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 3798.111725] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3799.362164] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3801.475278] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3801.568935] Lustre: lustre-OST0000: new disk, initializing [ 3801.570613] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3801.572742] Lustre: 151960:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3801.595901] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3803.363290] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 3806.317177] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3807.668583] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3810.295697] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3810.299241] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3810.325780] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3810.785662] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 3812.049705] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3812.118741] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3815.393838] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3815.394334] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3815.395656] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3815.395659] Lustre: Skipped 1 previous similar message [ 3815.397456] Lustre: Skipped 1 previous similar message [ 3815.405963] LustreError: Skipped 1 previous similar message [ 3818.825876] Lustre: server umount lustre-OST0000 complete [ 3820.513643] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3820.514272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3820.517616] Lustre: Skipped 1 previous similar message [ 3820.521916] Lustre: Skipped 2 previous similar messages [ 3825.633335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3825.634986] Lustre: Skipped 1 previous similar message [ 3834.336090] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3834.397038] Lustre: server umount lustre-MDT0000 complete [ 3835.600205] LustreError: 150212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780387586 with bad export cookie 11812862482553924425 [ 3835.603230] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3835.603741] LustreError: 150212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3835.710488] Lustre: server umount lustre-MDT0001 complete [ 3836.917393] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 04:06:27 (1780387587) [ 3837.432779] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 3838.002387] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 04:06:28 (1780387588) [ 3840.412849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3840.574972] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3841.825859] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 3844.652973] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3845.982778] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 3847.591640] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3848.770698] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3850.904826] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3852.710465] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 3854.383239] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3864.999738] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.8@tcp (stopping) [ 3865.001548] Lustre: Skipped 2 previous similar messages [ 3865.571819] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3865.574875] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3865.577224] Lustre: Skipped 2 previous similar messages [ 3866.208165] Lustre: server umount lustre-MDT0000 complete [ 3867.467164] LustreError: 154225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780387618 with bad export cookie 11812862482553925293 [ 3867.470680] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3867.471322] LustreError: 154225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3870.110448] LustreError: 154241:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3870.117868] LustreError: 154241:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3870.689049] LustreError: 154240:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3872.736972] LustreError: 154239:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3873.645586] Lustre: server umount lustre-MDT0001 complete [ 3875.749618] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3880.033176] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3880.116072] LustreError: 157743:0:(llog.c:1646:llog_backup()) MGC192.168.204.108@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 3880.121051] Lustre: 157743:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.108@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 3889.565697] LustreError: 157748:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3894.685702] LustreError: 157750:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3905.949466] LustreError: 157750:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3905.956182] LustreError: 157750:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 3925.985358] LustreError: 157750:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3925.992828] LustreError: 157750:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 14 previous similar messages [ 3940.960299] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 3940.962026] Lustre: 157743:0:(mgc_request.c:1901:mgc_process_log()) MGC192.168.204.108@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 3961.245330] LustreError: 157769:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3961.250722] LustreError: 157769:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 21 previous similar messages [ 3981.798833] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3981.801384] Lustre: Skipped 2 previous similar messages [ 3983.081372] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 3985.971316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3986.916020] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3987.936963] LustreError: 142064:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000009de71b6d x1866871557160192/t0(0) o250->MGC192.168.204.108@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3989.327558] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 3991.120558] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3991.667753] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3992.791681] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3993.061564] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3997.440463] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 4 sec [ 4003.298727] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4003.299344] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4003.299558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4003.299562] Lustre: Skipped 6 previous similar messages [ 4003.302230] Lustre: Skipped 2 previous similar messages [ 4004.114894] Lustre: server umount lustre-MDT0000 complete [ 4005.332104] LustreError: 154226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780387756 with bad export cookie 11812862482553931733 [ 4005.334882] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4005.337337] LustreError: 154226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4005.452137] Lustre: server umount lustre-MDT0001 complete [ 4016.859493] Lustre: server umount lustre-OST0000 complete [ 4023.422142] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [ 4026.705138] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 4037.414368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4037.498578] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4037.508678] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4037.543115] Lustre: lustre-MDT0000: new disk, initializing [ 4037.569949] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4038.742869] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 4042.754345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4042.790283] Lustre: 162558:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4042.882321] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4044.119769] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 4047.044629] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4048.165533] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4050.330745] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4050.424603] Lustre: 164299:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4050.453499] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4050.456333] Lustre: Skipped 3 previous similar messages [ 4051.063694] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4051.101358] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4052.244399] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 4055.225755] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4056.512834] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4056.577557] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4057.781246] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4057.843645] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4060.488615] Lustre: server umount lustre-OST0000 complete [ 4061.666320] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4061.667162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4061.670891] Lustre: Skipped 2 previous similar messages [ 4061.672860] Lustre: Skipped 2 previous similar messages [ 4071.928069] Lustre: server umount lustre-MDT0000 complete [ 4073.136149] LustreError: 162549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780387824 with bad export cookie 11812862482553948533 [ 4073.139235] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4073.141067] LustreError: 162549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4083.725108] Lustre: server umount lustre-MDT0001 complete [ 4086.656662] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 04:10:36 (1780387836) [ 4088.716568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4088.844829] LustreError: 166559:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4088.850241] LustreError: 166559:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 22 previous similar messages [ 4090.040157] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 4092.747775] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4094.093290] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 4095.896256] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4097.139058] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4099.359175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4101.358521] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 4103.120158] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4122.811487] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 04:11:12 (1780387872) [ 4124.130255] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4124.131277] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4124.136511] Lustre: Skipped 1 previous similar message [ 4129.250361] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4129.252307] Lustre: Skipped 8 previous similar messages [ 4129.876989] Lustre: server umount lustre-OST0000 complete [ 4133.857017] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4137.232791] Lustre: server umount lustre-MDT0000 complete [ 4138.436179] LustreError: 166541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780387889 with bad export cookie 11812862482553949401 [ 4138.438899] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4138.440598] LustreError: 166541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4138.549449] Lustre: server umount lustre-MDT0001 complete [ 4142.014775] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 4142.979575] Key type lgssc unregistered [ 4143.108396] LNet: 169946:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4143.110742] LNetError: 169946:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4143.123293] LNet: Removed LNI 192.168.204.108@tcp [ 4143.383287] Key type .llcrypt unregistered [ 4143.384264] Key type ._llcrypt unregistered [ 4150.600192] Key type ._llcrypt registered [ 4150.601227] Key type .llcrypt registered [ 4150.633973] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 4151.033221] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4151.042345] alg: No test for adler32 (adler32-zlib) [ 4151.905416] Lustre: Lustre: Build Version: 2.17.53_57_g7d1c020 [ 4151.990833] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [ 4153.576160] Key type lgssc registered [ 4153.931347] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4156.484122] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4156.489987] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4171.936317] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4177.056190] LustreError: 171149:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.108@tcp: failed processing log, type 4: rc = -110 [ 4182.176397] LustreError: 171149:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 4182.181146] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 4182.183599] LustreError: MGC192.168.204.108@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 4182.187859] LustreError: 171149:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-OST0000: -22 [ 4182.191241] LustreError: 171149:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -22 [ 4182.193534] LustreError: 171149:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 4182.219341] Lustre: server umount lustre-OST0000 complete [ 4182.221158] LustreError: 171149:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 4185.006378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4186.149566] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4187.470897] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 4190.297218] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4190.428602] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4191.725818] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 4193.511629] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4194.683076] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4196.799391] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4196.835210] LustreError: 173241:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4196.840360] LustreError: 173241:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4196.863812] LustreError: 173236:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 4196.867049] Lustre: 173236:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.204.108@tcp: invalid local config log lustre-OST0000: rc = -22 [ 4196.907441] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4198.673067] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 4200.407805] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4206.050516] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4206.052434] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4206.052565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4206.057055] Lustre: Skipped 2 previous similar messages [ 4207.252156] Lustre: server umount lustre-MDT0000 complete [ 4208.524920] LustreError: 171629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780387959 with bad export cookie 17432225050436271935 [ 4208.527321] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4208.528590] LustreError: 171629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4208.636327] Lustre: server umount lustre-MDT0001 complete [ 4219.162966] Lustre: server umount lustre-OST0000 complete [ 4221.948249] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [ 4225.158298] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 4236.988721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4237.074380] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4237.084669] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4237.119715] Lustre: lustre-MDT0000: new disk, initializing [ 4237.143438] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4237.149553] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4238.406461] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 4242.445371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4242.487170] Lustre: 176609:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4242.510157] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4242.512297] Lustre: Skipped 1 previous similar message [ 4242.543557] Lustre: lustre-MDT0001: new disk, initializing [ 4242.573754] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4242.584866] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4242.588778] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4243.839675] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 4246.840286] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4248.096528] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4250.360663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4250.458797] Lustre: lustre-OST0000: new disk, initializing [ 4250.461045] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4250.464626] Lustre: 178350:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4252.431624] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 4255.503271] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4256.847220] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4259.838400] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4259.842489] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4259.875717] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4259.977851] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 4261.305889] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4261.377735] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4264.930693] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4264.931921] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4264.934288] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4264.937342] Lustre: Skipped 1 previous similar message [ 4264.939398] Lustre: Skipped 2 previous similar messages [ 4268.116604] Lustre: server umount lustre-OST0000 complete [ 4270.049588] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4270.050284] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4270.053562] Lustre: Skipped 1 previous similar message [ 4270.055695] Lustre: Skipped 1 previous similar message [ 4275.168972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4275.170934] Lustre: Skipped 2 previous similar messages [ 4275.413233] Lustre: server umount lustre-MDT0000 complete [ 4276.640063] LustreError: 177386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780388027 with bad export cookie 17432225050436272628 [ 4276.643518] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4276.643996] LustreError: 177386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4276.751174] Lustre: server umount lustre-MDT0001 complete [ 4279.621456] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 04:13:49 (1780388029) [ 4285.330870] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 4286.307163] Key type lgssc unregistered [ 4286.420388] LNet: 181148:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4286.422858] LNetError: 181148:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4286.431251] LNet: Removed LNI 192.168.204.108@tcp [ 4286.692127] Key type .llcrypt unregistered [ 4286.693178] Key type ._llcrypt unregistered [ 4293.512622] Key type ._llcrypt registered [ 4293.513983] Key type .llcrypt registered [ 4293.551323] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 4293.956779] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4294.019925] alg: No test for adler32 (adler32-zlib) [ 4294.880086] Lustre: Lustre: Build Version: 2.17.53_57_g7d1c020 [ 4294.962885] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [ 4296.544163] Key type lgssc registered [ 4296.871809] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4299.278819] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4299.284991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4300.435533] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4301.737901] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 4304.487054] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4304.604768] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4305.922252] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 4307.598253] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4308.747574] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4311.118801] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4311.183307] Lustre: *** cfs_fail_loc=131e, val=0*** [ 4311.184541] LustreError: 183960:0:(llog.c:1687:llog_backup()) MGC192.168.204.108@tcp: failed to backup log lustre-OST0000: rc = -28 [ 4311.187028] Lustre: 183960:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.204.108@tcp: can't backup local config lustre-OST0000: rc = -28 [ 4311.220418] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4313.003447] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 4314.667410] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4319.715806] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4319.716749] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4319.719777] Lustre: Skipped 2 previous similar messages [ 4319.722880] Lustre: Skipped 2 previous similar messages [ 4321.437966] Lustre: server umount lustre-MDT0000 complete [ 4322.660115] LustreError: 182306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780388073 with bad export cookie 148910393885638442 [ 4322.662896] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4322.665554] LustreError: 182306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4322.774575] Lustre: server umount lustre-MDT0001 complete [ 4333.269563] Lustre: server umount lustre-OST0000 complete [ 4336.373328] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [ 4339.845795] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 4351.368196] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4351.457840] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4351.470580] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4351.510273] Lustre: lustre-MDT0000: new disk, initializing [ 4351.534693] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4351.542162] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4352.776875] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 4356.667202] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4356.708516] Lustre: 187328:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4356.729244] Lustre: *** cfs_fail_loc=131e, val=0*** [ 4356.730728] Lustre: Skipped 1 previous similar message [ 4356.732644] LustreError: 188095:0:(llog.c:1687:llog_backup()) MGC192.168.204.108@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 4356.735433] LustreError: 188095:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 4356.738151] Lustre: 188095:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.108@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 4356.746745] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4356.748487] Lustre: Skipped 1 previous similar message [ 4356.784307] Lustre: lustre-MDT0001: new disk, initializing [ 4356.815684] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4356.825479] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4356.828971] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4358.024692] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 4360.985706] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4362.121134] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4364.222040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4364.307355] Lustre: *** cfs_fail_loc=131e, val=0*** [ 4364.309142] Lustre: Skipped 1 previous similar message [ 4364.311159] LustreError: 189047:0:(llog.c:1687:llog_backup()) MGC192.168.204.108@tcp: failed to backup log lustre-OST0000: rc = -28 [ 4364.315092] LustreError: 189047:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 4364.318064] Lustre: 189047:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.108@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 4364.322520] Lustre: 189047:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 4364.331943] Lustre: lustre-OST0000: new disk, initializing [ 4364.333601] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4364.336867] Lustre: 189071:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4366.232054] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 4368.311304] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4368.314837] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4368.346888] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4369.375702] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4370.711693] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4370.785272] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4372.100268] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4372.185043] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4373.476651] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4373.476925] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4373.480942] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4373.482216] LustreError: Skipped 1 previous similar message [ 4374.867971] Lustre: server umount lustre-OST0000 complete [ 4377.568812] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4377.571852] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4377.576020] Lustre: Skipped 1 previous similar message [ 4377.577677] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4377.580565] Lustre: Skipped 1 previous similar message [ 4382.365111] Lustre: server umount lustre-MDT0000 complete [ 4383.626173] LustreError: 187323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780388134 with bad export cookie 148910393885639142 [ 4383.628987] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4383.630469] LustreError: 187323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4383.756372] Lustre: server umount lustre-MDT0001 complete [ 4386.759750] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 04:15:36 (1780388136) [ 4392.067914] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 4395.103316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4395.224781] LustreError: 192025:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4395.231142] LustreError: 192025:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4395.237914] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4395.239846] Lustre: Skipped 1 previous similar message [ 4396.446442] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 4399.107040] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4400.439466] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 4401.389658] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4403.416733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4405.231859] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 4408.022652] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4408.087794] Lustre: lustre-OST0001: new disk, initializing [ 4408.090077] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4408.093398] Lustre: 194280:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4408.165819] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4408.168873] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4408.206397] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4409.988108] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 4415.390757] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4419.984502] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4420.576609] Lustre: DEBUG MARKER: ADD OST3 [ 4422.186608] loop: module loaded [ 4422.204370] loop0: detected capacity change from 0 to 204800000 [ 4422.320958] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4422.324431] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4422.329168] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4424.304186] loop0: detected capacity change from 0 to 204800000 [ 4426.145834] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4426.205732] Lustre: lustre-OST0003: new disk, initializing [ 4426.207404] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 4426.210427] Lustre: 195991:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4426.249493] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 4426.251741] Lustre: Skipped 3 previous similar messages [ 4427.266619] Lustre: DEBUG MARKER: STOP OST3 [ 4427.848187] Lustre: *** cfs_fail_loc=2109, val=2*** [ 4427.882540] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 4427.886294] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 4427.897869] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 4427.924540] Lustre: Failing over lustre-OST0003 [ 4427.958284] Lustre: server umount lustre-OST0003 complete [ 4428.123840] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 4429.857107] LustreError: 196391:0:(osp_precreate.c:1281:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 4432.105756] Lustre: DEBUG MARKER: START OST3 again [ 4432.805064] LustreError: 193479:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.204.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4433.380834] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4433.387094] Lustre: Skipped 1 previous similar message [ 4434.183456] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4434.256103] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 4435.619091] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4436.032379] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 4436.198853] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4436.199035] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4449.544757] Lustre: Failing over lustre-OST0003 [ 4449.585992] Lustre: server umount lustre-OST0003 complete [ 4449.766220] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4449.768131] LustreError: 193477:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4449.771226] Lustre: Skipped 2 previous similar messages [ 4449.778326] LustreError: 193477:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4454.886612] LustreError: 193478:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4454.891932] LustreError: 193478:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4460.007174] LustreError: 193478:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4460.012521] LustreError: 193478:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4463.012133] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4463.013806] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4463.016044] Lustre: Skipped 3 previous similar messages [ 4463.019451] Lustre: Skipped 5 previous similar messages [ 4468.194625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4468.197725] Lustre: Skipped 3 previous similar messages [ 4468.198187] LustreError: 193479:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4468.203588] LustreError: 193479:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4476.384089] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4476.447548] Lustre: server umount lustre-MDT0000 complete [ 4477.640198] LustreError: 195086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780388228 with bad export cookie 148910393885640010 [ 4477.643563] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4477.644138] LustreError: 195086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4478.434846] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4478.436659] Lustre: Skipped 5 previous similar messages [ 4483.886944] Lustre: server umount lustre-MDT0001 complete [ 4485.240571] Lustre: server umount lustre-OST0000 complete [ 4486.523916] Lustre: server umount lustre-OST0001 complete [ 4488.681873] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [ 4491.510279] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 4503.350919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4503.471126] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4503.484906] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4503.529247] Lustre: lustre-MDT0000: new disk, initializing [ 4503.555827] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4503.558009] Lustre: Skipped 1 previous similar message [ 4503.564633] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4504.813185] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 4508.804108] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4508.851280] Lustre: 201332:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4508.951692] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4510.150107] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 4513.060641] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4514.230480] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4516.453814] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4516.548878] Lustre: 203074:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4518.433270] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 4518.522933] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4518.556987] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4521.460425] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4522.736946] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4522.805408] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4524.018790] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4524.090236] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4527.011260] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4527.012453] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4527.015108] Lustre: Skipped 3 previous similar messages [ 4527.016956] Lustre: Skipped 2 previous similar messages [ 4530.764861] Lustre: server umount lustre-OST0000 complete [ 4538.071954] Lustre: server umount lustre-MDT0000 complete [ 4539.235362] LustreError: 201326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780388290 with bad export cookie 148910393885642264 [ 4539.237392] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4539.243177] LustreError: 201326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4539.351490] Lustre: server umount lustre-MDT0001 complete [ 4540.456306] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 04:18:10 (1780388290) [ 4545.198846] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [ 4548.177767] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 4559.142446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4559.220220] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4559.230967] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4559.233359] Lustre: Skipped 3 previous similar messages [ 4559.269388] Lustre: lustre-MDT0000: new disk, initializing [ 4559.270622] Lustre: Skipped 2 previous similar messages [ 4559.294052] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4559.296421] Lustre: Skipped 2 previous similar messages [ 4560.459686] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 4564.409358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4564.454744] Lustre: 207494:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4564.553941] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4565.824599] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 4568.859828] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4570.120211] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4572.426638] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4572.525796] Lustre: 209235:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4572.553779] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4572.556047] Lustre: Skipped 4 previous similar messages [ 4574.172086] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4574.384124] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 4577.415446] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4578.758766] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4578.823947] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4580.119809] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4580.195114] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4584.417621] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4584.418486] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4584.420616] LustreError: Skipped 1 previous similar message [ 4584.425141] Lustre: Skipped 2 previous similar messages [ 4584.427267] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4584.429491] Lustre: Skipped 6 previous similar messages [ 4595.168123] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4595.230807] Lustre: server umount lustre-OST0000 complete [ 4602.592582] Lustre: server umount lustre-MDT0000 complete [ 4603.822314] LustreError: 207484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780388354 with bad export cookie 148910393885643146 [ 4603.824726] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4603.825632] LustreError: 207484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4603.938935] Lustre: server umount lustre-MDT0001 complete [ 4606.726018] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4606.898889] LustreError: 211310:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4606.905640] LustreError: 211310:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 4608.167429] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 4611.006513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4612.571554] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 4614.470254] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4615.740251] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4618.045056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4620.166543] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 4622.013107] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4641.762129] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4644.949280] Lustre: server umount lustre-OST0000 complete [ 4646.880986] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4646.883607] LustreError: Skipped 1 previous similar message [ 4652.443397] Lustre: server umount lustre-MDT0000 complete [ 4653.025203] LustreError: 211310:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4653.031108] LustreError: 211310:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4653.701425] LustreError: 211292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780388404 with bad export cookie 148910393885644028 [ 4653.704763] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4653.705441] LustreError: 211292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4653.853282] Lustre: server umount lustre-MDT0001 complete [ 4757.111494] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 4758.244470] Key type lgssc unregistered [ 4758.370587] LNet: 214638:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4758.374459] LNetError: 214638:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4758.384357] LNet: Removed LNI 192.168.204.108@tcp [ 4758.705133] Key type .llcrypt unregistered [ 4758.707156] Key type ._llcrypt unregistered [ 4767.103606] Key type ._llcrypt registered [ 4767.105754] Key type .llcrypt registered [ 4767.149582] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [ 4773.480637] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 4773.876500] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4773.886265] alg: No test for adler32 (adler32-zlib) [ 4774.738817] Lustre: Lustre: Build Version: 2.17.53_57_g7d1c020 [ 4774.821515] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [ 4776.400199] Key type lgssc registered [ 4776.771076] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4788.632930] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4788.640370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4789.744683] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4789.755927] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4789.790473] Lustre: lustre-MDT0000: new disk, initializing [ 4789.816118] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4789.825348] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4791.221547] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 4795.601299] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4795.655370] Lustre: 217927:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4795.681518] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4795.684969] Lustre: Skipped 1 previous similar message [ 4795.726606] Lustre: lustre-MDT0001: new disk, initializing [ 4795.761335] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4795.775627] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4795.778635] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4797.196963] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 4800.258517] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4801.359463] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4803.594494] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4803.727797] Lustre: lustre-OST0000: new disk, initializing [ 4803.730761] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 4803.730980] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4803.737585] Lustre: 219665:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4803.795468] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4805.743729] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 4808.966556] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4809.196099] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4809.198679] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4809.214177] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4810.307626] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4810.377123] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4811.592975] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4811.658891] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4814.309381] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4814.309979] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4814.310769] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4814.311536] LustreError: Skipped 1 previous similar message [ 4814.315764] Lustre: Skipped 1 previous similar message [ 4818.325528] Lustre: server umount lustre-OST0000 complete [ 4821.472971] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4821.475360] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4821.480187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4821.483479] Lustre: Skipped 1 previous similar message [ 4824.545309] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4824.545855] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4825.628974] Lustre: server umount lustre-MDT0000 complete [ 4826.779293] LustreError: 217919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780388577 with bad export cookie 6647199147218874240 [ 4826.783282] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4826.783729] LustreError: 217919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4826.911807] Lustre: server umount lustre-MDT0001 complete [ 4832.799117] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 4836.408269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4836.561803] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4837.923824] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 4840.731871] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4840.868352] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4842.147724] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 4843.120860] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4845.387589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4847.267050] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 4850.020020] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4850.080856] Lustre: lustre-OST0001: new disk, initializing [ 4850.082762] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4850.085373] Lustre: 224685:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4850.116229] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4850.119448] Lustre: Skipped 1 previous similar message [ 4851.146655] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4851.151100] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4851.195499] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4852.106427] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 4857.523303] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4861.249806] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4864.909455] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 04:23:35 (1780388615) [ 4866.531969] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4866.535676] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4866.536691] Lustre: Skipped 3 previous similar messages [ 4866.538647] Lustre: Skipped 3 previous similar messages [ 4866.539980] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4871.710547] Lustre: server umount lustre-MDT0000 complete [ 4872.971467] LustreError: 222414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780388623 with bad export cookie 6647199147218875080 [ 4872.975859] LustreError: 222414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4872.976513] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4873.079271] Lustre: server umount lustre-MDT0001 complete [ 4884.994109] Lustre: server umount lustre-OST0000 complete [ 4896.474620] Lustre: server umount lustre-OST0001 complete [ 4898.532659] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [ 4901.823481] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 4913.733183] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4913.825648] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4913.839949] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4913.877835] Lustre: lustre-MDT0000: new disk, initializing [ 4913.899805] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4913.906239] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4915.191716] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 4919.284911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4919.323593] Lustre: 228569:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4919.436803] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4920.727453] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 4924.359887] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4925.775316] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4928.198718] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4928.297088] Lustre: lustre-OST0000: new disk, initializing [ 4928.299331] Lustre: Skipped 1 previous similar message [ 4928.301321] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4928.304250] Lustre: Skipped 2 previous similar messages [ 4928.306463] Lustre: 230311:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4929.914829] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4929.917281] Lustre: Skipped 1 previous similar message [ 4929.919200] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4929.952628] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4930.197812] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 4933.209462] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4934.593721] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4934.666880] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4936.059436] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4936.156410] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4940.259043] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4940.260594] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4940.261636] LustreError: Skipped 1 previous similar message [ 4940.266089] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4940.269350] Lustre: Skipped 5 previous similar messages [ 4942.929797] Lustre: server umount lustre-OST0000 complete [ 4945.379313] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4950.366673] Lustre: server umount lustre-MDT0000 complete [ 4950.497173] LustreError: 228581:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4950.501983] LustreError: 228581:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4951.603510] LustreError: 229349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780388702 with bad export cookie 6647199147218877173 [ 4951.607057] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4951.607481] LustreError: 229349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4951.719662] Lustre: server umount lustre-MDT0001 complete [ 4954.580805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4954.760321] LustreError: 232379:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4954.786060] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4954.789154] Lustre: Skipped 2 previous similar messages [ 4956.161341] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 4959.016277] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4960.452928] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 4962.225073] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4963.392114] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4965.603798] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4967.562922] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 4969.379271] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5113.314358] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5113.314908] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5113.316652] LustreError: Skipped 1 previous similar message [ 5113.317205] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5113.321366] Lustre: Skipped 4 previous similar messages [ 5113.328578] Lustre: Skipped 3 previous similar messages [ 5117.134443] Lustre: server umount lustre-OST0000 complete [ 5124.501127] Lustre: server umount lustre-MDT0000 complete [ 5125.714254] LustreError: 232359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780388876 with bad export cookie 6647199147218878055 [ 5125.717161] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5125.720082] LustreError: 232359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5125.833929] Lustre: server umount lustre-MDT0001 complete [ 5232.129571] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 5233.060233] Key type lgssc unregistered [ 5233.187347] LNet: 235631:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5233.189880] LNetError: 235631:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5233.198265] LNet: Removed LNI 192.168.204.108@tcp [ 5233.508131] Key type .llcrypt unregistered [ 5233.509779] Key type ._llcrypt unregistered [ 5242.004319] Key type ._llcrypt registered [ 5242.005325] Key type .llcrypt registered [ 5242.056108] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [ 5248.568376] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 5249.072466] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5249.086168] alg: No test for adler32 (adler32-zlib) [ 5249.948735] Lustre: Lustre: Build Version: 2.17.53_57_g7d1c020 [ 5250.044697] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [ 5251.640152] Key type lgssc registered [ 5252.033961] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5266.577324] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 5269.895510] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5269.901661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5270.990879] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5271.002658] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5271.037275] Lustre: lustre-MDT0000: new disk, initializing [ 5271.059980] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5271.065946] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5272.378512] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 5276.547665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5276.598792] Lustre: 239620:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5276.623792] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5276.625863] Lustre: Skipped 1 previous similar message [ 5276.675328] Lustre: lustre-MDT0001: new disk, initializing [ 5276.712814] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5276.725437] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5276.729462] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5278.100354] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 5280.419984] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5282.757638] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5282.885533] Lustre: lustre-OST0000: new disk, initializing [ 5282.887128] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5282.890162] Lustre: 241216:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5282.926226] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5284.903778] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 5285.561976] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5285.565190] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5285.610410] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5289.231045] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5289.298703] Lustre: lustre-OST0001: new disk, initializing [ 5289.301380] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5289.304673] Lustre: 242077:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5289.343530] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5291.398379] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 5293.575145] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5293.580244] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5293.614294] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5296.770226] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5303.339610] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5307.361066] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5307.363855] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5307.368470] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5308.898438] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5308.899745] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5308.903615] Lustre: Skipped 1 previous similar message [ 5308.905773] Lustre: Skipped 1 previous similar message [ 5312.167563] Lustre: server umount lustre-MDT0000 complete [ 5313.387189] LustreError: 239612:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780389064 with bad export cookie 15294221168757157685 [ 5313.391351] LustreError: 239612:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5313.391408] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5313.513537] Lustre: server umount lustre-MDT0001 complete [ 5325.003571] Lustre: server umount lustre-OST0000 complete [ 5336.478888] Lustre: server umount lustre-OST0001 complete [ 5341.076492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5344.213749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5347.023539] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 5350.307331] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 5356.431022] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 5359.633995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5359.639523] Lustre: lustre-MDT0000: reset Object Index mappings [ 5359.796037] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5360.935153] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 5363.541945] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5363.547513] Lustre: lustre-MDT0001: reset Object Index mappings [ 5364.844194] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 5365.754425] Lustre: 248149:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5367.795365] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5367.901280] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5367.904469] Lustre: Skipped 1 previous similar message [ 5369.625492] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 5369.955814] LustreError: 248502:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5369.960818] LustreError: 248502:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5369.962394] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5372.486067] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5373.611927] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 5374.486591] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 5377.646906] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5378.903810] Lustre: 249980:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 5384.166817] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5384.167389] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5384.170105] Lustre: Skipped 2 previous similar messages [ 5384.170357] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5384.170363] Lustre: Skipped 2 previous similar messages [ 5386.208235] Lustre: server umount lustre-MDT0000 complete [ 5387.500219] LustreError: 247020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780389138 with bad export cookie 15294221168757164461 [ 5387.503309] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5387.504455] LustreError: 247020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5387.626879] Lustre: server umount lustre-MDT0001 complete [ 5398.470545] Lustre: server umount lustre-OST0000 complete [ 5410.009560] Lustre: server umount lustre-OST0001 complete [ 5416.212684] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 04:32:46 (1780389166) [ 5421.515405] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [ 5424.813300] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 5436.864020] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5436.961408] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5436.975231] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5437.017094] Lustre: lustre-MDT0000: new disk, initializing [ 5437.043969] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5437.045971] Lustre: Skipped 1 previous similar message [ 5437.051983] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5438.363131] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 5442.588713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5442.630682] Lustre: 254190:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5442.733424] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5444.003942] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 5447.071865] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5448.230261] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5450.425504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5450.531682] Lustre: lustre-OST0000: new disk, initializing [ 5450.533710] Lustre: Skipped 1 previous similar message [ 5450.535565] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5450.539327] Lustre: Skipped 2 previous similar messages [ 5450.542854] Lustre: 255930:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5451.709068] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5451.711485] Lustre: Skipped 1 previous similar message [ 5451.713473] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5451.753990] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5452.517611] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 5455.595429] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5456.968919] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5457.052165] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5458.406157] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5458.480814] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5461.986257] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5461.987059] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5461.989555] LustreError: Skipped 1 previous similar message [ 5461.990623] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5461.995473] Lustre: Skipped 1 previous similar message [ 5461.997920] Lustre: Skipped 3 previous similar messages [ 5465.241381] Lustre: server umount lustre-OST0000 complete [ 5467.107269] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5467.108079] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5467.112745] Lustre: Skipped 1 previous similar message [ 5467.114941] Lustre: Skipped 1 previous similar message [ 5472.671669] Lustre: server umount lustre-MDT0000 complete [ 5473.908278] LustreError: 254181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780389224 with bad export cookie 15294221168757166526 [ 5473.912075] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5473.912225] LustreError: 254181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5474.031077] Lustre: server umount lustre-MDT0001 complete [ 5480.068406] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 5483.525093] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5483.671336] LustreError: 258691:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5483.688134] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5483.691319] Lustre: Skipped 2 previous similar messages [ 5485.029904] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 5487.974181] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5489.570661] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 5490.616338] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5492.865084] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5494.797940] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 5497.635401] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5497.700988] Lustre: lustre-OST0001: new disk, initializing [ 5497.702593] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5497.704867] Lustre: 260947:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5499.005156] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5499.008613] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5499.047401] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5499.660594] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 5504.099662] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5505.270573] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5507.125046] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 5507.129869] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 5509.602026] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5509.605325] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5509.608029] Lustre: Skipped 3 previous similar messages [ 5509.610208] Lustre: Skipped 5 previous similar messages [ 5515.042361] Lustre: server umount lustre-MDT0000 complete [ 5516.262364] LustreError: 259410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780389267 with bad export cookie 15294221168757167415 [ 5516.266098] LustreError: 259410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5516.266564] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5516.381925] Lustre: server umount lustre-MDT0001 complete [ 5528.068325] Lustre: server umount lustre-OST0000 complete [ 5539.621212] Lustre: server umount lustre-OST0001 complete [ 5544.961147] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 5548.336988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5548.473296] LustreError: 263962:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5548.493440] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5548.495891] Lustre: Skipped 3 previous similar messages [ 5549.746721] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 5552.528618] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5553.998308] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 5554.957068] Lustre: 265068:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5557.175208] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5558.372585] LustreError: 265421:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5558.377691] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 5559.154689] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 5561.991282] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5563.123395] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 5563.934960] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 5567.289737] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5568.448646] Lustre: 266906:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 5570.122598] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 5570.128424] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 5573.270526] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 04:35:23 (1780389323) [ 5588.449175] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5588.451527] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5588.455550] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5588.457383] Lustre: Skipped 5 previous similar messages [ 5590.886112] Lustre: server umount lustre-MDT0000 complete [ 5592.108434] LustreError: 263944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780389343 with bad export cookie 15294221168757173722 [ 5592.114075] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5594.081501] LustreError: 263958:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5594.085902] LustreError: 263958:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5602.209598] LustreError: 265444:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5602.214298] LustreError: 265444:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5606.368122] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5606.462357] Lustre: server umount lustre-MDT0001 complete [ 5622.240172] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5622.297520] Lustre: server umount lustre-OST0000 complete [ 5630.451858] Lustre: server umount lustre-OST0001 complete [ 5633.810604] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [ 5639.369741] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 5661.972726] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 5666.095752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5666.218672] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5666.233851] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5666.274797] Lustre: lustre-MDT0000: new disk, initializing [ 5666.315122] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5668.166658] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 5672.967559] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5673.021605] Lustre: 271092:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5673.202652] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5674.907211] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 5680.546157] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5680.668557] Lustre: 272690:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5680.707838] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5680.710389] Lustre: Skipped 5 previous similar messages [ 5681.852779] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5682.996809] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 5687.588226] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5688.884541] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5689.895012] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 5694.781557] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5700.873780] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 04:37:30 (1780389450) [ 5704.163052] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5704.164296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5704.167846] Lustre: Skipped 7 previous similar messages [ 5704.170779] Lustre: Skipped 10 previous similar messages [ 5704.173525] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5707.872853] Lustre: server umount lustre-MDT0000 complete [ 5709.163172] LustreError: 271870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780389460 with bad export cookie 15294221168757181604 [ 5709.168117] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5709.168369] LustreError: 271870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 5709.279488] Lustre: server umount lustre-MDT0001 complete [ 5720.770191] Lustre: server umount lustre-OST0000 complete [ 5731.193652] Lustre: server umount lustre-OST0001 complete [ 5734.539013] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5734.743939] LustreError: 275767:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5736.288443] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 5739.376853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5740.950418] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 5742.924550] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5744.368423] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5747.258117] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5749.625060] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 5751.668933] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5767.070686] LustreError: 277930:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5767.077656] LustreError: 277930:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 5768.391571] loop1: detected capacity change from 0 to 204800000 [ 5768.475187] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5768.480558] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5768.488658] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5770.477174] loop1: detected capacity change from 0 to 204800000 [ 5772.459809] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5772.496864] LustreError: 275757:0:(mgs_llog.c:673:mgs_find_or_make_fsdb_nolock()) fs19442: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 5772.503052] LustreError: 275757:0:(mgs_llog.c:731:mgs_check_index()) Can't get db for fs19442 [ 5772.505648] LustreError: 275757:0:(mgs_handler.c:517:mgs_target_reg()) fs19442-MDT0001: Error checking target index: rc = -13 [ 5772.509487] LustreError: fs19442-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 5772.513758] LustreError: 279167:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -13 [ 5772.517130] LustreError: 279167:0:(tgt_mount.c:2084:server_put_super()) no obd fs19442-MDT0001 [ 5772.519408] LustreError: 279167:0:(tgt_mount.c:116:server_deregister_mount()) fs19442-MDT0001 not registered [ 5772.533618] Lustre: server umount fs19442-MDT0001 complete [ 5772.535799] LustreError: 279167:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 5775.623390] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5775.656370] Lustre: Setting parameter fs19442-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs19442-MDT0001 [ 5775.661288] Lustre: Skipped 2 previous similar messages [ 5775.679138] Lustre: srv-fs19442-MDT0001: No data found on store. Initialize space. [ 5775.681639] Lustre: Skipped 4 previous similar messages [ 5775.719676] Lustre: fs19442-MDT0001: new disk, initializing [ 5775.722049] Lustre: Skipped 3 previous similar messages [ 5775.761216] Lustre: srv-fs19442-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 5775.764715] Lustre: 279631:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs19442-MDT0001-mdtlov: can't create new llog: rc = -115 [ 5775.768191] LustreError: 279631:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs19442-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 5777.096253] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 5780.193279] Lustre: server umount fs19442-MDT0001 complete [ 5784.976608] loop1: detected capacity change from 0 to 204800000 [ 5787.259357] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5787.375691] Lustre: 281179:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs19442-MDT0001-mdtlov: can't create new llog: rc = -115 [ 5787.378966] LustreError: 281179:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs19442-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 5788.680381] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 5790.423101] Lustre: server umount fs19442-MDT0001 complete [ 5795.811717] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5795.811989] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5795.815358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5795.818152] Lustre: Skipped 2 previous similar messages [ 5800.021922] Lustre: server umount lustre-MDT0000 complete [ 5803.222790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5803.295465] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5804.939371] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 5807.805471] loop1: detected capacity change from 0 to 204800000 [ 5808.623171] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5809.967188] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5810.098017] Lustre: 283590:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs19442-MDT0001-mdtlov: can't create new llog: rc = -115 [ 5810.101209] LustreError: 283590:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs19442-MDT0001-osd: get update log duration 1, retries 0, failed: rc = -115 [ 5811.570867] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 5813.545420] Lustre: server umount fs19442-MDT0001 complete [ 5821.409137] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5827.227437] Lustre: server umount lustre-OST0000 complete [ 5834.786594] Lustre: server umount lustre-MDT0000 complete [ 5836.153472] LustreError: 275748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780389587 with bad export cookie 15294221168757184278 [ 5836.156377] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5836.157964] LustreError: 275748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5836.297520] Lustre: server umount lustre-MDT0001 complete [ 5840.122490] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 5841.301631] Key type lgssc unregistered [ 5841.439437] LNet: 285894:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5841.442460] LNetError: 285894:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5841.451325] LNet: Removed LNI 192.168.204.108@tcp [ 5841.776182] Key type .llcrypt unregistered [ 5841.778222] Key type ._llcrypt unregistered [ 5849.475693] Key type ._llcrypt registered [ 5849.477120] Key type .llcrypt registered [ 5849.517854] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 5849.952269] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5849.971450] alg: No test for adler32 (adler32-zlib) [ 5850.850558] Lustre: Lustre: Build Version: 2.17.53_57_g7d1c020 [ 5850.945419] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [ 5852.536202] Key type lgssc registered [ 5852.953775] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5856.021909] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5856.029821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5857.226329] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5858.654309] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 5861.858344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5862.013396] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5863.375161] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 5865.058843] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5866.269279] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5868.680980] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5868.830486] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5870.785336] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 5871.912511] LustreError: 289233:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5871.920190] LustreError: 289233:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5872.534440] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5877.218675] LustreError: 288730:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5882.340833] LustreError: 288728:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5882.346895] LustreError: 288728:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5887.459496] LustreError: 289233:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5887.463785] LustreError: 289233:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5889.207497] loop1: detected capacity change from 0 to 204800000 [ 5889.327355] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5889.331220] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5889.334761] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5891.064723] loop1: detected capacity change from 0 to 204800000 [ 5892.579934] LustreError: 289233:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5892.589030] LustreError: 289233:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5893.066631] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5893.126494] Lustre: fs19887-OST0000: new disk, initializing [ 5893.129256] Lustre: srv-fs19887-OST0000: No data found on store. Initialize space. [ 5893.131626] Lustre: 290540:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5893.154280] Lustre: fs19887-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5895.139796] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 5898.553291] Lustre: server umount fs19887-OST0000 complete [ 5902.800520] loop1: detected capacity change from 0 to 204800000 [ 5902.823132] LustreError: 289287:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5902.832433] LustreError: 289287:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 5904.791870] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5904.869813] Lustre: fs19887-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5906.863490] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 5909.830045] loop2: detected capacity change from 0 to 204800000 [ 5909.912079] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5909.916127] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5909.921184] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5911.671524] loop2: detected capacity change from 0 to 204800000 [ 5913.581077] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5913.609568] LustreError: fs19887-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [ 5913.612807] LustreError: fs19887-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 5913.615315] LustreError: 293541:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -13 [ 5913.617308] LustreError: 293541:0:(tgt_mount.c:2084:server_put_super()) no obd fs19887-OST0001 [ 5913.619039] LustreError: 293541:0:(tgt_mount.c:116:server_deregister_mount()) fs19887-OST0001 not registered [ 5913.629978] Lustre: server umount fs19887-OST0001 complete [ 5913.631419] LustreError: 293541:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 5917.368297] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5917.417234] Lustre: fs19887-OST0001: new disk, initializing [ 5917.419688] Lustre: srv-fs19887-OST0001: No data found on store. Initialize space. [ 5917.422974] Lustre: 294093:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5917.450542] Lustre: fs19887-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5919.134720] LustreError: 289233:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5919.142086] LustreError: 289233:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 5919.498375] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 5923.311521] Lustre: server umount fs19887-OST0000 complete [ 5926.739133] Lustre: server umount fs19887-OST0001 complete [ 5930.791681] loop1: detected capacity change from 0 to 204800000 [ 5932.716512] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5934.709575] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 5937.150631] loop2: detected capacity change from 0 to 204800000 [ 5939.050381] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5939.109732] Lustre: fs19887-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5939.112070] Lustre: Skipped 1 previous similar message [ 5941.054485] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 5944.011247] Lustre: server umount fs19887-OST0000 complete [ 5947.131207] Lustre: server umount fs19887-OST0001 complete [ 5954.019154] LustreError: 288730:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5954.020174] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5954.020367] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5954.022897] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5954.026796] LustreError: 288730:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 19 previous similar messages [ 5954.030574] Lustre: Skipped 1 previous similar message [ 5959.141888] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5959.145429] Lustre: Skipped 2 previous similar messages [ 5964.260273] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5964.262357] Lustre: Skipped 1 previous similar message [ 5964.263351] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [ 5964.266458] LustreError: Skipped 1 previous similar message [ 5965.280184] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5965.337693] Lustre: server umount lustre-OST0000 complete [ 5969.378034] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5969.378182] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5969.379067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5969.387834] Lustre: Skipped 1 previous similar message [ 5972.768637] Lustre: server umount lustre-MDT0000 complete [ 5974.040379] LustreError: 287852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780389724 with bad export cookie 17845177858832921810 [ 5974.044642] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5974.046045] LustreError: 287852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5974.187947] Lustre: server umount lustre-MDT0001 complete [ 5977.677219] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 5978.631120] Key type lgssc unregistered [ 5978.749340] LNet: 299940:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5978.751503] LNetError: 299940:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5978.761292] LNet: Removed LNI 192.168.204.108@tcp [ 5979.047117] Key type .llcrypt unregistered [ 5979.048441] Key type ._llcrypt unregistered [ 5988.898574] Key type ._llcrypt registered [ 5988.899815] Key type .llcrypt registered [ 5988.938035] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [ 5994.862872] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 5995.320706] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5995.332876] alg: No test for adler32 (adler32-zlib) [ 5996.196145] Lustre: Lustre: Build Version: 2.17.53_57_g7d1c020 [ 5996.277853] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [ 5997.856128] Key type lgssc registered [ 5998.226893] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6010.330922] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6010.337513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6011.422560] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6011.434374] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6011.467155] Lustre: lustre-MDT0000: new disk, initializing [ 6011.491189] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6011.498959] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6012.865504] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 6017.378099] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6017.421645] Lustre: 303733:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6017.445127] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6017.447530] Lustre: Skipped 1 previous similar message [ 6017.486756] Lustre: lustre-MDT0001: new disk, initializing [ 6017.523665] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6017.537386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6017.540837] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6018.899636] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 6022.089566] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6023.261066] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6025.567580] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6025.679061] Lustre: lustre-OST0000: new disk, initializing [ 6025.680826] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6025.684129] Lustre: 305477:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6025.715653] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6026.878475] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6026.882922] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6026.932878] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6027.818236] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 6030.922581] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6031.614940] Lustre: Mounted lustre-client [ 6031.951200] Lustre: Unmounted lustre-client [ 6036.962672] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6036.963660] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6036.965791] Lustre: Skipped 1 previous similar message [ 6040.928504] Lustre: server umount lustre-OST0000 complete [ 6043.105192] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6043.107429] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6043.112075] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6043.114776] Lustre: Skipped 1 previous similar message [ 6047.201577] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6047.202442] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6048.480703] Lustre: server umount lustre-MDT0000 complete [ 6049.764183] LustreError: 303727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780389800 with bad export cookie 12846423200333970227 [ 6049.767624] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6049.769171] LustreError: 303727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6049.901749] Lustre: server umount lustre-MDT0001 complete [ 6053.738080] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 6054.771482] Key type lgssc unregistered [ 6088.480160] LNet: 1 local NIs in recovery (showing 1): 192.168.204.108@tcp [ 6088.672409] LNet: 307505:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6088.675055] LNetError: 307505:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6162.406471] LNet: Removed LNI 192.168.204.108@tcp [ 6162.727743] Key type .llcrypt unregistered [ 6162.729041] Key type ._llcrypt unregistered [ 6169.466884] Key type ._llcrypt registered [ 6169.468268] Key type .llcrypt registered [ 6169.515154] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [ 6179.047698] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 6179.724138] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6179.737637] alg: No test for adler32 (adler32-zlib) [ 6180.783396] Lustre: Lustre: Build Version: 2.17.53_57_g7d1c020 [ 6180.929962] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [ 6182.560151] Key type lgssc registered [ 6183.500774] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6205.358329] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 04:45:54 (1780389954) [ 6212.828934] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6212.879267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6214.271709] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6214.330714] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6214.486860] Lustre: lustre-MDT0000: new disk, initializing [ 6214.613081] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6214.642637] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6219.753569] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 6229.011290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6229.139096] Lustre: 310675:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6229.225518] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6229.230304] Lustre: Skipped 1 previous similar message [ 6229.336807] Lustre: lustre-MDT0001: new disk, initializing [ 6229.436931] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6229.478220] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6229.485511] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6232.885183] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 6238.556794] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6241.399598] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6246.502075] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6246.671074] Lustre: lustre-OST0000: new disk, initializing [ 6246.673661] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6246.677701] Lustre: 312419:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6246.723437] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6250.654705] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 6255.055928] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6255.624692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6255.631272] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6255.684725] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6265.825915] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6265.827793] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6265.829291] LustreError: Skipped 1 previous similar message [ 6265.829871] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6265.834960] Lustre: Skipped 1 previous similar message [ 6267.310669] Lustre: server umount lustre-OST0000 complete [ 6270.433598] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6270.438615] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6270.447819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6270.450170] Lustre: Skipped 1 previous similar message [ 6271.291308] Lustre: server umount lustre-MDT0000 complete [ 6273.152180] LustreError: 310669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780390024 with bad export cookie 8853982074715858365 [ 6273.154097] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6273.159024] LustreError: 310669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6273.372472] Lustre: server umount lustre-MDT0001 complete [ 6278.915482] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 6280.466287] Key type lgssc unregistered [ 6280.649105] LNet: 314045:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6280.654382] LNetError: 314045:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6280.664264] LNet: Removed LNI 192.168.204.108@tcp [ 6281.173140] Key type .llcrypt unregistered [ 6281.175867] Key type ._llcrypt unregistered [ 6295.163579] Key type ._llcrypt registered [ 6295.165353] Key type .llcrypt registered [ 6295.227899] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 6295.747829] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6295.787806] alg: No test for adler32 (adler32-zlib) [ 6296.708796] Lustre: Lustre: Build Version: 2.17.53_57_g7d1c020 [ 6296.818596] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [ 6298.424588] Key type lgssc registered [ 6299.015726] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6302.770174] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6302.780216] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6304.020946] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6305.948379] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 6310.154500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6310.362159] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6312.156562] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 6314.799793] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6316.388336] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6319.639202] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6319.816046] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6322.578266] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 6325.273085] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6335.193506] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6335.957377] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6339.553786] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6339.555484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6339.565794] Lustre: Skipped 2 previous similar messages [ 6339.567811] Lustre: Skipped 1 previous similar message [ 6342.064280] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.8@tcp (stopping) [ 6343.909827] Lustre: server umount lustre-MDT0000 complete [ 6344.674273] LustreError: 315425:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6344.684515] LustreError: 315425:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6345.951786] LustreError: 315405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780390096 with bad export cookie 3038998806336073968 [ 6345.956506] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6345.960820] LustreError: 315405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6347.170806] LustreError: 315420:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6347.181596] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.8@tcp (stopping) [ 6349.796996] LustreError: 315422:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6349.797692] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6349.821162] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6352.263260] Lustre: server umount lustre-MDT0001 complete [ 6360.496610] Lustre: server umount lustre-OST0000 complete [ 6364.897947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6366.954324] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 6368.267284] Lustre: 319150:0:(mgs_llog.c:1798:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.108@tcp [ 6369.242267] Lustre: server umount MGS complete [ 6372.925380] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6373.151496] LustreError: 319722:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6373.160517] LustreError: 319722:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6373.187171] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6374.947323] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 6378.472645] LustreError: 319723:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6378.624877] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6378.814713] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6380.587593] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 6384.661140] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6387.495775] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 6390.292889] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6391.107641] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [ 6392.956622] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6404.298516] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 10 sec [ 6408.675712] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6408.680457] Lustre: Skipped 1 previous similar message [ 6408.681308] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6408.686234] Lustre: Skipped 2 previous similar messages [ 6411.882849] Lustre: server umount lustre-OST0000 complete [ 6413.794879] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6413.803744] Lustre: Skipped 1 previous similar message [ 6418.913796] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6418.919310] Lustre: Skipped 3 previous similar messages [ 6419.812549] Lustre: server umount lustre-MDT0000 complete [ 6421.435553] LustreError: 319704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780390172 with bad export cookie 3038998806336074794 [ 6421.437957] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6421.441874] LustreError: 319704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6421.591408] Lustre: server umount lustre-MDT0001 complete [ 6426.191145] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 6427.467196] Key type lgssc unregistered [ 6427.616545] LNet: 322915:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6427.621301] LNetError: 322915:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6427.630633] LNet: Removed LNI 192.168.204.108@tcp [ 6428.013225] Key type .llcrypt unregistered [ 6428.015198] Key type ._llcrypt unregistered [ 6436.873354] Key type ._llcrypt registered [ 6436.874701] Key type .llcrypt registered [ 6436.925843] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [ 6444.610446] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 6445.318556] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6445.352176] alg: No test for adler32 (adler32-zlib) [ 6446.276190] Lustre: Lustre: Build Version: 2.17.53_57_g7d1c020 [ 6446.397318] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [ 6448.000168] Key type lgssc registered [ 6448.559428] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6466.000539] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6466.013395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6467.141099] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6467.161717] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6467.204310] Lustre: lustre-MDT0000: new disk, initializing [ 6467.254825] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6467.267437] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6469.209527] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 6475.068807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6475.162841] Lustre: 326017:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6475.202278] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6475.205644] Lustre: Skipped 1 previous similar message [ 6475.259446] Lustre: lustre-MDT0001: new disk, initializing [ 6475.326110] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6475.353864] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6475.363169] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6477.435089] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 6482.007596] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6483.932878] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6487.526714] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6487.676702] Lustre: lustre-OST0000: new disk, initializing [ 6487.679268] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6487.683151] Lustre: 327760:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6487.727875] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6488.788659] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6488.794373] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6488.871424] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6490.658420] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 6495.585231] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6498.335452] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6498.481493] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6501.231206] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6501.356950] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6504.418833] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6504.420164] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6504.424473] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6504.431000] Lustre: Skipped 1 previous similar message [ 6504.452126] LustreError: Skipped 1 previous similar message [ 6508.656346] Lustre: server umount lustre-OST0000 complete [ 6511.073513] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6511.078582] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6511.089537] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6511.094301] Lustre: Skipped 1 previous similar message [ 6514.657963] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6514.658396] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6516.769410] Lustre: server umount lustre-MDT0000 complete [ 6518.774110] LustreError: 326010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780390269 with bad export cookie 11229315746027342276 [ 6518.777213] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6518.780883] LustreError: 326010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6518.954649] Lustre: server umount lustre-MDT0001 complete [ 6523.728520] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 04:51:13 (1780390273) [ 6532.122976] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [ 6536.612692] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 6553.629607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6553.789482] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6553.810384] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6553.862514] Lustre: lustre-MDT0000: new disk, initializing [ 6553.909313] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6553.923651] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6555.885362] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 6561.296443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6561.348660] Lustre: 332332:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6561.377979] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6561.382381] Lustre: Skipped 1 previous similar message [ 6561.408088] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6561.411364] Lustre: Skipped 1 previous similar message [ 6561.424789] Lustre: lustre-MDT0001: new disk, initializing [ 6561.470559] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6561.486137] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6561.490738] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6563.337995] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 6567.563084] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6569.420889] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6572.797725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6572.937756] Lustre: lustre-OST0000: new disk, initializing [ 6572.941729] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6572.946442] Lustre: 334071:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6573.002759] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6575.701889] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 6579.201195] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6579.208273] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6579.262993] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6579.863588] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6596.664704] Lustre: server umount lustre-OST0000 complete [ 6599.650613] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6599.651840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6599.657103] Lustre: Skipped 1 previous similar message [ 6599.661500] Lustre: Skipped 1 previous similar message [ 6608.909422] Lustre: server umount lustre-MDT0000 complete [ 6610.598650] LustreError: 332323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780390361 with bad export cookie 11229315746027343151 [ 6610.600835] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6610.606265] LustreError: 332323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6620.202692] Lustre: server umount lustre-MDT0001 complete [ 6624.990713] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 6626.319933] Key type lgssc unregistered [ 6626.473762] LNet: 336090:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6626.478080] LNetError: 336090:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6626.485891] LNet: Removed LNI 192.168.204.108@tcp [ 6626.877495] Key type .llcrypt unregistered [ 6626.879326] Key type ._llcrypt unregistered [ 6669.645081] Key type ._llcrypt registered [ 6669.647205] Key type .llcrypt registered [ 6669.689485] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 6669.832477] Key type .llcrypt unregistered [ 6669.833970] Key type ._llcrypt unregistered [ 6704.020740] Key type ._llcrypt registered [ 6704.024503] Key type .llcrypt registered [ 6704.082651] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 6704.241789] Key type .llcrypt unregistered [ 6704.245731] Key type ._llcrypt unregistered [ 6717.799214] Key type ._llcrypt registered [ 6717.800834] Key type .llcrypt registered [ 6717.856569] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 6718.000506] Key type .llcrypt unregistered [ 6718.003086] Key type ._llcrypt unregistered [ 6731.156543] Key type ._llcrypt registered [ 6731.158106] Key type .llcrypt registered [ 6731.205629] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 6731.343054] Key type .llcrypt unregistered [ 6731.344815] Key type ._llcrypt unregistered [ 6765.583139] Key type ._llcrypt registered [ 6765.584597] Key type .llcrypt registered [ 6765.636514] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 6765.787388] Key type .llcrypt unregistered [ 6765.789196] Key type ._llcrypt unregistered [ 6778.691090] Key type ._llcrypt registered [ 6778.692911] Key type .llcrypt registered [ 6778.738880] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 6778.853281] Key type .llcrypt unregistered [ 6778.855141] Key type ._llcrypt unregistered [ 6792.345516] Key type ._llcrypt registered [ 6792.347314] Key type .llcrypt registered [ 6792.402184] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 6792.523549] Key type .llcrypt unregistered [ 6792.526085] Key type ._llcrypt unregistered [ 6806.242568] Key type ._llcrypt registered [ 6806.244262] Key type .llcrypt registered [ 6806.294790] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 6806.414701] Key type .llcrypt unregistered [ 6806.417269] Key type ._llcrypt unregistered [ 6819.689185] Key type ._llcrypt registered [ 6819.690919] Key type .llcrypt registered [ 6819.741827] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 6819.874286] Key type .llcrypt unregistered [ 6819.876844] Key type ._llcrypt unregistered [ 6834.242713] Key type ._llcrypt registered [ 6834.244373] Key type .llcrypt registered [ 6834.295769] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 6834.465321] Key type .llcrypt unregistered [ 6834.467725] Key type ._llcrypt unregistered [ 6863.341126] Key type ._llcrypt registered [ 6863.342627] Key type .llcrypt registered [ 6863.395061] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 6863.529679] Key type .llcrypt unregistered [ 6863.531591] Key type ._llcrypt unregistered [ 6877.044497] Key type ._llcrypt registered [ 6877.046580] Key type .llcrypt registered [ 6877.103170] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 6877.239974] Key type .llcrypt unregistered [ 6877.241956] Key type ._llcrypt unregistered [ 6890.434381] Key type ._llcrypt registered [ 6890.436198] Key type .llcrypt registered [ 6890.491479] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 6890.624402] Key type .llcrypt unregistered [ 6890.626912] Key type ._llcrypt unregistered [ 6914.773218] Key type ._llcrypt registered [ 6914.774924] Key type .llcrypt registered [ 6914.835895] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [ 6922.190404] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 6922.733111] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6922.744107] alg: No test for adler32 (adler32-zlib) [ 6923.633973] Lustre: Lustre: Build Version: 2.17.53_57_g7d1c020 [ 6923.735793] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [ 6925.336175] Key type lgssc registered [ 6925.845260] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6940.507221] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6940.520893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6941.641278] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [ 6941.658387] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [ 6941.702431] Lustre: tst250fs-MDT0000: new disk, initializing [ 6941.739134] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6941.749618] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6943.527740] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 6948.706325] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6948.759048] Lustre: 355762:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6948.787546] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [ 6948.790050] Lustre: Skipped 1 previous similar message [ 6948.827887] Lustre: tst250fs-MDT0001: new disk, initializing [ 6948.868527] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6948.888071] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6948.892865] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6950.437898] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 6952.966370] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6955.708516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6955.827194] Lustre: tst250fs-OST0000: new disk, initializing [ 6955.830124] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [ 6955.834147] Lustre: 357360:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6955.875333] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6958.306885] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 6960.648098] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6960.653510] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6960.709815] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6963.432578] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6963.508807] Lustre: tst250fs-OST0001: new disk, initializing [ 6963.511338] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [ 6963.515371] Lustre: 358215:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6963.556764] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6965.831240] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 6968.708396] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [ 6969.626260] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6969.630360] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6969.690323] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6980.066924] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6980.069466] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 6980.072146] Lustre: Skipped 3 previous similar messages [ 6980.074546] Lustre: Skipped 2 previous similar messages [ 6985.186055] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 6985.188609] Lustre: Skipped 3 previous similar messages [ 6990.306440] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 6994.400227] Lustre: tst250fs-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6994.487534] Lustre: server umount tst250fs-MDT0000 complete [ 6996.022695] LustreError: 355753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780390746 with bad export cookie 17434482666449849281 [ 6996.025912] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6996.030336] LustreError: 355753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6996.168223] Lustre: server umount tst250fs-MDT0001 complete [ 7007.757882] Lustre: server umount tst250fs-OST0000 complete [ 7019.563255] Lustre: server umount tst250fs-OST0001 complete [ 7026.483574] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [ 7029.721381] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 7041.542263] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 04:59:51 (1780390791) [ 7042.130900] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [ 7048.284621] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [ 7052.374765] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 7064.982970] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 6773 sec ========== 05:00:14 (1780390814) [ 7065.732473] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 05:00:15 (1780390815) === [ 7067.130478] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 05:00:17 (1780390817) === [ 7083.096134] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 7084.305585] Key type lgssc unregistered [ 7084.444371] LNet: 367251:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7084.447458] LNetError: 367251:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7084.456458] LNet: Removed LNI 192.168.204.108@tcp [ 7084.824179] Key type .llcrypt unregistered [ 7084.825711] Key type ._llcrypt unregistered