[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 3.0.0 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-2.fc40 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 485115218 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f53f0-0x000f53ff] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5200 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1D87 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1C23 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001BE3 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1C97 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1D27 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1D5F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1c23-0xbffe1c96] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1c22] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1c97-0xbffe1d26] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1d27-0xbffe1d5e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1d5f-0xbffe1d86] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.002399] x2apic enabled [ 0.003010] Switched APIC routing to physical x2apic. [ 0.004015] kvm-guest: setup PV IPIs [ 0.007582] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009014] pid_max: default: 32768 minimum: 301 [ 0.010138] LSM: Security Framework initializing [ 0.012073] Yama: becoming mindful. [ 0.013041] SELinux: Initializing. [ 0.014071] *** VALIDATE selinux *** [ 0.021000] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025535] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027015] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028123] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029116] *** VALIDATE tmpfs *** [ 0.031277] *** VALIDATE proc *** [ 0.033179] *** VALIDATE cgroup *** [ 0.034013] *** VALIDATE cgroup2 *** [ 0.035271] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037003] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039027] Spectre V2 : User space: Vulnerable [ 0.040012] Speculative Store Bypass: Vulnerable [ 0.042943] debug: unmapping init [mem 0xffffffff95c59000-0xffffffff95c60fff] [ 0.045182] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046741] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047031] ... version: 2 [ 0.048015] ... bit width: 48 [ 0.049012] ... generic registers: 4 [ 0.050014] ... value mask: 0000ffffffffffff [ 0.051015] ... max period: 00007fffffffffff [ 0.052015] ... fixed-purpose events: 3 [ 0.053012] ... event mask: 000000070000000f [ 0.054296] rcu: Hierarchical SRCU implementation. [ 0.056376] smp: Bringing up secondary CPUs ... [ 0.057550] x86: Booting SMP configuration: [ 0.058030] .... node #0, CPUs: #1 #2 #3 [ 0.061705] smp: Brought up 1 node, 4 CPUs [ 0.063016] smpboot: Max logical packages: 1 [ 0.064017] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.238374] node 0 deferred pages initialised in 171ms [ 0.242012] devtmpfs: initialized [ 0.243264] x86/mm: Memory block size: 128MB [ 0.245771] gcov: version magic: 0x41383552 [ 0.247275] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.248109] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.249266] pinctrl core: initialized pinctrl subsystem [ 0.250230] [ 0.250762] ************************************************************* [ 0.251018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.252017] ** ** [ 0.253016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.254016] ** ** [ 0.255021] ** This means that this kernel is built to expose internal ** [ 0.256018] ** IOMMU data structures, which may compromise security on ** [ 0.257020] ** your system. ** [ 0.258022] ** ** [ 0.259035] ** If you see this message and you are not debugging the ** [ 0.260020] ** kernel, report this immediately to your vendor! ** [ 0.261022] ** ** [ 0.262017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.263020] ************************************************************* [ 0.264653] NET: Registered protocol family 16 [ 0.265457] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.266075] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.267069] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.268489] cpuidle: using governor menu [ 0.270801] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.273963] PCI: Using configuration type 1 for base access [ 0.276156] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.287108] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.288057] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.290085] cryptd: max_cpu_qlen set to 1000 [ 0.293303] ACPI: Added _OSI(Module Device) [ 0.295024] ACPI: Added _OSI(Processor Device) [ 0.297021] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.298027] ACPI: Added _OSI(Processor Aggregator Device) [ 0.302584] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.309397] ACPI: Interpreter enabled [ 0.311064] ACPI: PM: (supports S0 S3 S4 S5) [ 0.312015] ACPI: Using IOAPIC for interrupt routing [ 0.314161] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.317630] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.327787] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.330066] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.332027] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.336129] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.340536] acpiphp: Slot [2] registered [ 0.343124] acpiphp: Slot [5] registered [ 0.344232] acpiphp: Slot [6] registered [ 0.346132] acpiphp: Slot [7] registered [ 0.347107] acpiphp: Slot [8] registered [ 0.348123] acpiphp: Slot [9] registered [ 0.350132] acpiphp: Slot [10] registered [ 0.351180] acpiphp: Slot [3] registered [ 0.353132] acpiphp: Slot [4] registered [ 0.354127] acpiphp: Slot [11] registered [ 0.356123] acpiphp: Slot [12] registered [ 0.357136] acpiphp: Slot [13] registered [ 0.358115] acpiphp: Slot [14] registered [ 0.360143] acpiphp: Slot [15] registered [ 0.362157] acpiphp: Slot [16] registered [ 0.363138] acpiphp: Slot [17] registered [ 0.365122] acpiphp: Slot [18] registered [ 0.366116] acpiphp: Slot [19] registered [ 0.368167] acpiphp: Slot [20] registered [ 0.369136] acpiphp: Slot [21] registered [ 0.371165] acpiphp: Slot [22] registered [ 0.372142] acpiphp: Slot [23] registered [ 0.374156] acpiphp: Slot [24] registered [ 0.376149] acpiphp: Slot [25] registered [ 0.377133] acpiphp: Slot [26] registered [ 0.379129] acpiphp: Slot [27] registered [ 0.380135] acpiphp: Slot [28] registered [ 0.382187] acpiphp: Slot [29] registered [ 0.384126] acpiphp: Slot [30] registered [ 0.385225] acpiphp: Slot [31] registered [ 0.387128] PCI host bridge to bus 0000:00 [ 0.388023] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.390029] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.393037] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.395035] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.398040] pci_bus 0000:00: root bus resource [mem 0x380000000000-0x38007fffffff window] [ 0.401048] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.403216] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.407230] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.410658] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.419516] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.424068] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.426028] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.431030] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.433023] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.435628] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.437710] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.439038] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.441870] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.445021] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.457025] pci 0000:00:02.0: reg 0x20: [mem 0x380000000000-0x380000003fff 64bit pref] [ 0.462022] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.467000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.473028] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.480028] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.496036] pci 0000:00:05.0: reg 0x20: [mem 0x380000004000-0x380000007fff 64bit pref] [ 0.504544] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.507022] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.513022] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.527028] pci 0000:00:06.0: reg 0x20: [mem 0x380000008000-0x38000000bfff 64bit pref] [ 0.536600] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.542025] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.549028] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.562031] pci 0000:00:07.0: reg 0x20: [mem 0x38000000c000-0x38000000ffff 64bit pref] [ 0.570000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.577017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.583017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.600031] pci 0000:00:08.0: reg 0x20: [mem 0x380000010000-0x380000013fff 64bit pref] [ 0.612083] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.618020] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.625020] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.639019] pci 0000:00:09.0: reg 0x20: [mem 0x380000014000-0x380000017fff 64bit pref] [ 0.651290] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.658015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.664027] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.676029] pci 0000:00:0a.0: reg 0x20: [mem 0x380000018000-0x38000001bfff 64bit pref] [ 0.688190] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.690444] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.693415] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.695390] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.698269] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.702210] iommu: Default domain type: Passthrough [ 0.704507] SCSI subsystem initialized [ 0.706173] ACPI: bus type USB registered [ 0.707155] usbcore: registered new interface driver usbfs [ 0.709123] usbcore: registered new interface driver hub [ 0.711153] usbcore: registered new device driver usb [ 0.713265] pps_core: LinuxPPS API ver. 1 registered [ 0.715020] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.718083] PTP clock support registered [ 0.720327] EDAC MC: Ver: 3.0.0 [ 0.723066] PCI: Using ACPI for IRQ routing [ 0.724891] NetLabel: Initializing [ 0.726012] NetLabel: domain hash size = 128 [ 0.727008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.729084] NetLabel: unlabeled traffic allowed by default [ 0.732157] vgaarb: loaded [ 0.734142] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.735012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.738348] clocksource: Switched to clocksource kvm-clock [ 0.836402] VFS: Disk quotas dquot_6.6.0 [ 0.837791] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.839842] *** VALIDATE ramfs *** [ 0.840883] *** VALIDATE hugetlbfs *** [ 0.842142] pnp: PnP ACPI init [ 0.843965] pnp: PnP ACPI: found 6 devices [ 0.866598] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.869242] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.870963] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.872697] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.874603] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.876505] pci_bus 0000:00: resource 8 [mem 0x380000000000-0x38007fffffff window] [ 0.878858] NET: Registered protocol family 2 [ 0.880958] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.884978] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.888138] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.893077] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.895920] TCP: Hash tables configured (established 65536 bind 65536) [ 0.898290] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.900740] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.902989] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.905484] NET: Registered protocol family 1 [ 0.913854] RPC: Registered named UNIX socket transport module. [ 0.915552] RPC: Registered udp transport module. [ 0.916855] RPC: Registered tcp transport module. [ 0.918101] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.920045] NET: Registered protocol family 44 [ 0.921875] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.924352] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.926062] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.927938] PCI: CLS 0 bytes, default 64 [ 0.929465] Unpacking initramfs... [ 2.332284] debug: unmapping init [mem 0xffff91cdfcc54000-0xffff91cdfffbffff] [ 2.336457] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.338921] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.341774] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.841739] Initialise system trusted keyrings [ 2.843189] Key type blacklist registered [ 2.844759] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.851956] zbud: loaded [ 2.854540] *** VALIDATE nfs *** [ 2.855680] *** VALIDATE nfs4 *** [ 2.856964] pstore: using deflate compression [ 2.860109] Platform Keyring initialized [ 2.959925] NET: Registered protocol family 38 [ 2.961930] Key type asymmetric registered [ 2.964212] Asymmetric key parser 'x509' registered [ 2.965968] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.969403] io scheduler mq-deadline registered [ 2.971232] io scheduler kyber registered [ 2.973028] io scheduler bfq registered [ 2.975052] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.978412] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.981377] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.984479] ACPI: Power Button [PWRF] [ 3.074685] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.163970] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.327439] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.417562] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.601562] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.629736] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.659591] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.664588] Non-volatile memory driver v1.3 [ 3.666388] Linux agpgart interface v0.103 [ 3.699735] virtio_blk virtio1: [vda] 133832 512-byte logical blocks (68.5 MB/65.3 MiB) [ 3.703107] vda: detected capacity change from 0 to 68521984 [ 3.728214] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.731218] vdb: detected capacity change from 0 to 1073741824 [ 3.752517] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.754925] vdc: detected capacity change from 0 to 2621440000 [ 3.771172] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.774077] vdd: detected capacity change from 0 to 2621440000 [ 3.793831] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.797479] vde: detected capacity change from 0 to 4294967296 [ 3.817059] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.820136] vdf: detected capacity change from 0 to 4294967296 [ 3.828133] libphy: Fixed MDIO Bus: probed [ 3.836418] usbcore: registered new interface driver usbserial_generic [ 3.839031] usbserial: USB Serial support registered for generic [ 3.841360] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.845570] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.847960] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.850567] mousedev: PS/2 mouse device common for all mice [ 3.853053] rtc_cmos 00:05: RTC can wake from S4 [ 3.856445] rtc_cmos 00:05: registered as rtc0 [ 3.858184] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.859242] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.865288] intel_pstate: CPU model not supported [ 3.868398] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.869164] hid: raw HID events driver (C) Jiri Kosina [ 3.875608] usbcore: registered new interface driver usbhid [ 3.877751] usbhid: USB HID core driver [ 3.877911] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.879280] drop_monitor: Initializing network drop monitor service [ 3.885308] Initializing XFRM netlink socket [ 3.887446] NET: Registered protocol family 10 [ 3.890889] Segment Routing with IPv6 [ 3.892512] NET: Registered protocol family 17 [ 3.895114] mpls_gso: MPLS GSO support [ 3.902173] RAS: Correctable Errors collector initialized. [ 3.904137] AVX version of gcm_enc/dec engaged. [ 3.905343] AES CTR mode by8 optimization enabled [ 3.984577] sched_clock: Marking stable (3984546984, 0)->(4965125696, -980578712) [ 3.987978] registered taskstats version 1 [ 3.990167] Loading compiled-in X.509 certificates [ 3.992515] zswap: loaded using pool lzo/zbud [ 4.017581] Key type big_key registered [ 4.031360] Key type encrypted registered [ 4.033299] ima: No TPM chip found, activating TPM-bypass! [ 4.035647] ima: Allocated hash algorithm: sha1 [ 4.038261] ima: No architecture policies found [ 4.040236] evm: Initialising EVM extended attributes: [ 4.041985] evm: security.selinux [ 4.043200] evm: security.ima [ 4.044366] evm: security.capability [ 4.045517] evm: HMAC attrs: 0x1 [ 4.048119] rtc_cmos 00:05: setting system clock to 2025-11-16 23:44:32 UTC (1763336672) [ 4.054376] debug: unmapping init [mem 0xffffffff96c03000-0xffffffff96dfffff] [ 4.057464] debug: unmapping init [mem 0xffffffff95982000-0xffffffff95c58fff] [ 4.066095] Write protecting the kernel read-only data: 28672k [ 4.069209] debug: unmapping init [mem 0xffffffff94003000-0xffffffff941fffff] [ 4.071579] debug: unmapping init [mem 0xffffffff94914000-0xffffffff949fffff] [ 4.105189] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.113969] systemd[1]: Detected virtualization kvm. [ 4.116415] systemd[1]: Detected architecture x86-64. [ 4.119304] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.166902] systemd[1]: No hostname configured. [ 4.168830] systemd[1]: Set hostname to . [ 4.173415] random: systemd: uninitialized urandom read (16 bytes read) [ 4.175827] systemd[1]: Initializing machine ID from random generator. [ 4.214362] random: ln: uninitialized urandom read (6 bytes read) [ 4.308981] random: systemd: uninitialized urandom read (16 bytes read) [ 4.310726] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 4.314596] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 4.318797] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local File Systems. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Reached target Local Encrypted Volumes. Starting Setup Virtual Console... Starting Journal Service... [ OK ] Reached target Timers. [ OK ] Reached target Initrd Root Device. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.928028] device-mapper: uevent: version 1.0.3 [ 4.930834] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook...[ 5.564957] virtio_net virtio0 ens2: renamed from eth0 [ 5.581222] random: fast init done [ 5.681063] scsi host0: ata_piix [ 5.705866] scsi host1: ata_piix [ 5.707614] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.710297] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.967127] dracut-initqueue[587]: RTNETLINK answers: File exists [ 10.550610] random: crng init done [ 10.551927] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.973337] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.171869] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.430961] SELinux: Disabled at runtime. [ 12.497992] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.508957] systemd[1]: Detected virtualization kvm. [ 12.511884] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.060643] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.064189] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.068297] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.071976] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.075222] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.087361] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.094125] systemd[1]: Reached target rpc_pipefs.target. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting Kernel Debug File System... [ OK ] Listening on udev Control Socket. [ OK ] Stopped target Switch Root. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-getty.slice. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on Process Core Dump Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Initrd Root File System. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting POSIX Message Queue File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ 13.227966] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Create list of required st…ce nodes for the current kernel... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Apply Kernel Variables... Mounting Huge Pages File System... Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Initrd File Systems. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 13.617412] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.863913] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.967214] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.167466] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.183780] EDAC sbridge: Ver: 1.1.2 [ 15.914419] Key type dns_resolver registered [ 16.221542] NFS: Registering the id_resolver key type [ 16.223767] Key type id_resolver registered [ 16.225136] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg116-server login: [ 45.805911] libcfs: loading out-of-tree module taints kernel. [ 45.828187] Key type ._llcrypt registered [ 45.829847] Key type .llcrypt registered [ 45.919468] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 57.320967] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 58.169771] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 58.178813] alg: No test for adler32 (adler32-zlib) [ 59.301734] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 59.803469] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 61.471168] Key type lgssc registered [ 62.585512] Lustre: Echo OBD driver; http://www.lustre.org/ [ 73.945969] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 74.912704] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 80.458541] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 86.274198] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 92.640913] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 104.580150] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 116.510747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 116.561989] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 116.577979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 117.809727] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 117.841538] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 117.921940] Lustre: lustre-MDT0000: new disk, initializing [ 118.014905] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 118.047405] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 121.860504] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 134.027693] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 134.095923] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 134.146396] Lustre: 6483:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 134.170240] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 134.172659] Lustre: Skipped 1 previous similar message [ 134.248834] Lustre: lustre-MDT0001: new disk, initializing [ 134.310096] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 134.336440] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 134.346645] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 137.577299] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 141.587987] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 148.610566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 148.672633] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 148.893377] Lustre: lustre-OST0000: new disk, initializing [ 148.899862] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 148.977532] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 153.614348] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 153.629298] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 153.699331] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 153.703392] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 165.985257] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 166.066513] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 166.173517] Lustre: lustre-OST0001: new disk, initializing [ 166.182295] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 166.244476] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 166.962666] hrtimer: interrupt took 8657189 ns [ 170.693370] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 175.649774] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 175.660757] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 175.719989] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 179.410199] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 185.266426] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 193.673637] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing check_logdir /tmp/testlogs/ [ 196.533855] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing yml_node [ 199.268687] Lustre: DEBUG MARKER: Client: 2.16.61.51 [ 200.693640] Lustre: DEBUG MARKER: MDS: 2.16.61.51 [ 202.164291] Lustre: DEBUG MARKER: OSS: 2.16.61.51 [ 203.000599] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Nov 16 18:47:49 EST 2025 [ 214.344499] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 215.395954] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 231.904515] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 231.905540] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 231.905907] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 231.920679] Lustre: Skipped 3 previous similar messages [ 237.029124] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 237.033056] Lustre: Skipped 6 previous similar messages [ 242.143791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 243.167962] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 243.278403] LustreError: 12184:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 243.414563] Lustre: server umount lustre-MDT0000 complete [ 247.539903] LustreError: 10249:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763336915 with bad export cookie 10212438658836191833 [ 247.545034] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 247.552806] LustreError: 10249:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 252.384198] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 252.384628] LustreError: 6490:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 252.393538] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 252.412146] LustreError: 6490:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 252.431976] Lustre: Skipped 4 previous similar messages [ 253.776662] LustreError: 12388:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 253.788747] LustreError: 12388:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 253.938020] Lustre: server umount lustre-MDT0001 complete [ 258.691160] LustreError: 12590:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 258.721903] LustreError: 12590:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 258.922843] Lustre: server umount lustre-OST0000 complete [ 263.994596] LustreError: 12790:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 263.997664] LustreError: 12790:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 264.201754] Lustre: server umount lustre-OST0001 complete [ 271.189907] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 279.572216] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 286.212169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 291.034622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 296.147161] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 300.780565] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 306.503609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 306.549970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 306.721340] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 306.750578] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 306.800918] Lustre: lustre-MDT0000: new disk, initializing [ 306.873115] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 306.897833] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 310.131596] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 318.479666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 318.529948] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 318.597318] Lustre: 14971:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 318.656775] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 318.659908] Lustre: Skipped 1 previous similar message [ 318.717889] Lustre: lustre-MDT0001: new disk, initializing [ 318.797423] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 318.836831] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 318.856441] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 322.254310] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 328.695720] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 331.489374] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 337.161235] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 337.211700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 337.478224] Lustre: lustre-OST0000: new disk, initializing [ 337.485326] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 337.548612] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 339.550232] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 339.570431] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 339.734836] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 343.533818] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 350.589820] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 354.382030] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 354.529101] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 358.768618] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 358.956488] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 365.541904] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 365.545496] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 365.549970] LustreError: Skipped 1 previous similar message [ 365.563859] Lustre: Skipped 1 previous similar message [ 365.565398] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 365.576567] Lustre: Skipped 1 previous similar message [ 375.265891] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 375.320339] LustreError: 17916:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 375.335086] LustreError: 17916:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 375.509642] Lustre: server umount lustre-OST0000 complete [ 380.386597] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 380.405878] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 380.422322] Lustre: Skipped 1 previous similar message [ 380.429447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 380.432117] Lustre: Skipped 2 previous similar messages [ 384.620205] LustreError: 18118:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 384.634805] LustreError: 18118:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 384.740985] Lustre: server umount lustre-MDT0000 complete [ 386.021215] LustreError: 14983:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 386.059127] LustreError: 14983:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 389.545489] LustreError: 15755:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763337057 with bad export cookie 10212438658836193366 [ 389.552743] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 389.567424] LustreError: 15755:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 390.117738] Lustre: server umount lustre-MDT0001 complete [ 396.598156] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 18:51:02 (1763337062) [ 412.291240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 412.679148] LustreError: 19456:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 412.749575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 416.127813] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 417.765838] LustreError: 19456:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 422.894564] LustreError: 19456:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 424.404358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 428.343741] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 433.070914] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 436.102871] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 437.966572] Lustre: Failing over lustre-MDT0000 [ 438.099290] LustreError: 20799:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 438.108865] LustreError: 20799:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 438.179282] Lustre: server umount lustre-MDT0000 complete [ 440.290306] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 440.299264] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 440.334932] Lustre: Skipped 1 previous similar message [ 440.341455] LustreError: 19456:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 440.369188] LustreError: 19456:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 455.715336] Lustre: 3628:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763337108/real 1763337108] req@ffff91cd44744e00 x1848992573043584/t0(0) o400->MGC192.168.201.116@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1763337124 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 455.763581] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 457.796631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 461.282193] LustreError: 19449:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 461.298967] LustreError: 19449:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 469.862627] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 469.869355] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 469.889312] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 470.401304] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 476.721977] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 480.224913] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 480.227991] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 480.269804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 480.272793] Lustre: Skipped 1 previous similar message [ 486.356812] LustreError: 21782:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 486.359738] LustreError: 21782:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 486.436029] Lustre: server umount lustre-MDT0000 complete [ 489.559316] LustreError: 19435:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763337158 with bad export cookie 10212438658836194780 [ 489.565710] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 489.794051] Lustre: server umount lustre-MDT0001 complete [ 497.660930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 497.985936] LustreError: 22460:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 498.002312] LustreError: 22460:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 498.071311] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 498.074518] Lustre: Skipped 2 previous similar messages [ 502.210461] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 509.954819] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 513.725364] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 518.417752] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 521.729232] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 523.497340] Lustre: Failing over lustre-MDT0000 [ 523.736667] Lustre: server umount lustre-MDT0000 complete [ 525.792774] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 525.797285] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 525.803827] Lustre: Skipped 1 previous similar message [ 542.552819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 542.749810] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 546.885870] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 548.324348] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 548.338128] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 548.341298] Lustre: Skipped 1 previous similar message [ 548.350748] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 553.413305] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 558.561579] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 558.565228] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 558.575839] Lustre: Skipped 1 previous similar message [ 558.578989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 558.582021] Lustre: Skipped 2 previous similar messages [ 562.870617] LustreError: 24782:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 562.883139] LustreError: 24782:0:(obd_class.h:479:obd_check_dev()) Skipped 19 previous similar messages [ 562.999660] Lustre: server umount lustre-MDT0000 complete [ 565.731787] LustreError: 22460:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 565.756738] LustreError: 22460:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 13 previous similar messages [ 566.025673] LustreError: 22439:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763337234 with bad export cookie 10212438658836195711 [ 566.030078] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 566.033271] LustreError: 22439:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 566.307218] Lustre: server umount lustre-MDT0001 complete [ 572.401174] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 18:53:58 (1763337238) [ 583.518388] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 590.320442] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 597.239905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 601.720958] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 605.898414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 610.975663] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 618.410892] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 625.871809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 625.921789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 626.083346] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 626.116389] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 626.164259] Lustre: lustre-MDT0000: new disk, initializing [ 626.227621] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 626.231689] Lustre: Skipped 2 previous similar messages [ 626.248715] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 628.709868] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 636.264062] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 636.327036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 636.424915] Lustre: 28453:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 636.622272] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 639.959086] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 646.073626] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 648.823763] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 655.474094] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 655.557876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 657.329210] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 657.379369] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 657.414921] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 657.423895] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 661.976511] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 669.908313] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 675.242800] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 698.343119] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 698.344310] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 698.345900] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 698.351470] Lustre: Skipped 2 previous similar messages [ 698.353422] Lustre: Skipped 2 previous similar messages [ 700.575574] LustreError: 31109:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 700.578710] LustreError: 31109:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 700.675724] Lustre: server umount lustre-OST0000 complete [ 710.560605] Lustre: server umount lustre-MDT0000 complete [ 713.426094] LustreError: 30206:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763337381 with bad export cookie 10212438658836196110 [ 713.427616] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 713.435319] LustreError: 30206:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 713.760945] Lustre: server umount lustre-MDT0001 complete [ 722.056533] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 724.125761] Key type lgssc unregistered [ 724.333434] LNet: 32045:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 724.339902] LNetError: 32045:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 724.362230] LNet: Removed LNI 192.168.201.116@tcp [ 725.014245] Key type .llcrypt unregistered [ 725.016173] Key type ._llcrypt unregistered [ 744.345956] Key type ._llcrypt registered [ 744.347713] Key type .llcrypt registered [ 744.452828] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 745.693686] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 745.748216] alg: No test for adler32 (adler32-zlib) [ 746.776359] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 746.977601] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 748.623202] Key type lgssc registered [ 749.782345] Lustre: Echo OBD driver; http://www.lustre.org/ [ 757.358726] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 757.383640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 758.840161] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 762.659614] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 769.004044] LustreError: 33438:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 769.035370] LustreError: 33438:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 770.023125] LustreError: 33438:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 771.799376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 772.242606] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 775.684681] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 780.742881] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 784.498353] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 791.319198] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 791.721489] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 796.397722] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 796.862646] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:970 to 0x280000402:1025) [ 801.506324] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 805.064287] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 818.454300] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 18:58:04 (1763337484) [ 820.213781] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 821.438380] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 826.159098] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 18:58:12 (1763337492) [ 833.104714] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 18:58:19 (1763337499) [ 834.071748] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 842.744791] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 842.754524] Lustre: Skipped 2 previous similar messages [ 844.851271] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 18:58:30 (1763337510) [ 847.058346] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 847.061811] Lustre: Skipped 2 previous similar messages [ 855.821250] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 855.825319] Lustre: Skipped 4 previous similar messages [ 868.641768] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 18:58:54 (1763337534) [ 870.842791] Lustre: *** cfs_fail_loc=131b, val=2*** [ 873.756721] Lustre: *** cfs_fail_loc=131b, val=2*** [ 881.344153] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 18:59:07 (1763337547) [ 882.927166] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 882.930637] Lustre: Skipped 6 previous similar messages [ 894.182343] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 18:59:20 (1763337560) [ 907.748308] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 907.751979] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 907.766694] Lustre: Skipped 1 previous similar message [ 912.866806] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 912.869677] Lustre: Skipped 2 previous similar messages [ 917.985297] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 917.996599] Lustre: Skipped 1 previous similar message [ 919.007208] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 919.075435] LustreError: 40007:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 919.195879] Lustre: server umount lustre-OST0000 complete [ 923.107746] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 923.113939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 923.117031] Lustre: Skipped 1 previous similar message [ 928.227941] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 928.232522] Lustre: Skipped 1 previous similar message [ 928.491947] LustreError: 40209:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 928.497892] LustreError: 40209:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 928.588863] Lustre: server umount lustre-MDT0000 complete [ 931.262249] LustreError: 33420:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763337599 with bad export cookie 11020699302144105162 [ 931.263189] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 931.268707] LustreError: 33420:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 931.375560] LustreError: 40410:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 931.379683] LustreError: 40410:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 931.520828] Lustre: server umount lustre-MDT0001 complete [ 940.871461] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 943.264637] Key type lgssc unregistered [ 943.526793] LNet: 40942:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 943.539972] LNetError: 40942:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 944.550334] LNet: Removed LNI 192.168.201.116@tcp [ 945.214936] Key type .llcrypt unregistered [ 945.218387] Key type ._llcrypt unregistered [ 958.254095] Key type ._llcrypt registered [ 958.255604] Key type .llcrypt registered [ 958.340249] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 969.277419] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 969.824636] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 969.870344] alg: No test for adler32 (adler32-zlib) [ 970.908402] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 971.190055] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 972.839324] Key type lgssc registered [ 973.790945] Lustre: Echo OBD driver; http://www.lustre.org/ [ 980.656257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 985.242603] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 991.089320] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 996.756242] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1003.911899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1003.998698] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1004.022205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1005.292282] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1005.348293] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1005.446967] Lustre: lustre-MDT0000: new disk, initializing [ 1005.554353] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1005.574363] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1009.217272] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1018.382713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1018.469647] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1018.539312] Lustre: 44061:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1018.598489] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1018.605036] Lustre: Skipped 1 previous similar message [ 1018.701141] Lustre: lustre-MDT0001: new disk, initializing [ 1018.838590] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1018.886126] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1018.900169] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1022.776957] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1032.235948] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1037.103502] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1044.555458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1044.617657] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1044.877715] Lustre: lustre-OST0000: new disk, initializing [ 1044.880948] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1044.981220] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1050.232668] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1050.245166] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1050.343583] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1050.638425] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1057.321543] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1061.161311] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1061.354674] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1064.860471] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1065.035613] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1071.072679] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1071.077174] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1071.089066] LustreError: Skipped 1 previous similar message [ 1071.093778] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1071.111411] Lustre: Skipped 1 previous similar message [ 1076.195549] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1076.208748] Lustre: Skipped 2 previous similar messages [ 1081.311166] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1081.320945] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1081.333577] Lustre: Skipped 1 previous similar message [ 1081.375138] LustreError: 47006:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1081.487370] Lustre: server umount lustre-OST0000 complete [ 1085.411526] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1085.418790] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1085.426127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1086.435968] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1090.634831] LustreError: 47209:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1090.641573] LustreError: 47209:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1090.720867] Lustre: server umount lustre-MDT0000 complete [ 1094.039817] LustreError: 44842:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763337762 with bad export cookie 4247505137009350065 [ 1094.045725] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1094.227552] LustreError: 47411:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1094.235427] LustreError: 47411:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1094.395956] Lustre: server umount lustre-MDT0001 complete [ 1102.160148] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 19:02:48 (1763337768) [ 1108.115757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1108.519497] LustreError: 48130:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1108.532271] LustreError: 48130:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1108.601517] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1112.105593] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1114.083564] LustreError: 48130:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1118.182081] LustreError: 48130:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1120.534617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1120.865648] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1124.674255] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1129.475534] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1132.715325] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1138.502848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1138.730888] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1143.687982] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1148.817460] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1157.551136] LustreError: 50327:0:(llog.c:441:llog_init_handle()) llog type is not specified! [ 1158.350969] Lustre: Modifying parameter general.timeout=1 in log params [ 1159.891060] Lustre: Modifying parameter general.timeout=3 in log params [ 1159.893474] Lustre: Skipped 1 previous similar message [ 1162.571144] Lustre: Modifying parameter general.timeout=6 in log params [ 1162.575613] Lustre: Skipped 2 previous similar messages [ 1167.104784] Lustre: Modifying parameter general.timeout=11 in log params [ 1167.107133] Lustre: Skipped 4 previous similar messages [ 1175.359037] Lustre: Modifying parameter general.timeout=20 in log params [ 1175.361495] Lustre: Skipped 8 previous similar messages [ 1191.425932] Lustre: Modifying parameter general.timeout=41 in log params [ 1191.430124] Lustre: Skipped 20 previous similar messages [ 1224.060166] Lustre: Modifying parameter general.timeout=85 in log params [ 1224.062982] Lustre: Skipped 43 previous similar messages [ 1266.850535] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 19:05:33 (1763337933) [ 1269.119858] Lustre: 57267:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1270.952600] Lustre: 57365:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1274.086097] Lustre: 57561:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1274.092220] Lustre: 57561:0:(mgs_llog.c:1346:mgs_modify_param()) Skipped 1 previous similar message [ 1288.304594] Lustre: Disabling parameter general.ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 in log params [ 1288.310708] Lustre: Skipped 60 previous similar messages [ 1302.079594] Lustre: 59129:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1302.097315] Lustre: 59129:0:(mgs_llog.c:1346:mgs_modify_param()) Skipped 2 previous similar messages [ 1302.118298] LustreError: 59129:0:(mgs_handler.c:1126:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1306.191500] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 19:06:12 (1763337972) [ 1320.417451] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1320.421254] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1320.439160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1320.448818] Lustre: Skipped 2 previous similar messages [ 1332.191140] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1332.280464] LustreError: 59830:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1332.288347] LustreError: 59830:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1332.400867] Lustre: server umount lustre-MDT0000 complete [ 1332.708512] LustreError: 48842:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1332.731444] LustreError: 48842:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1335.617259] LustreError: 57017:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763338004 with bad export cookie 4247505137009350947 [ 1335.618461] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1335.629399] LustreError: 57017:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 1336.106899] Lustre: server umount lustre-MDT0001 complete [ 1350.406592] LustreError: 60233:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1350.419067] LustreError: 60233:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1350.584423] Lustre: server umount lustre-OST0000 complete [ 1358.446369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1362.098183] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1365.330409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1368.844695] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1375.857481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1376.274142] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1376.600062] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1380.341907] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1389.649083] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1389.881959] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1389.920468] Lustre: 61687:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1394.471621] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1400.286867] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1404.120425] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1410.874600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1411.089150] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1411.383353] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1411.413911] Lustre: Skipped 1 previous similar message [ 1416.948786] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1422.614486] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1424.591946] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1424.607773] Lustre: Skipped 7 previous similar messages [ 1424.651382] Lustre: 63919:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1434.084855] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1434.099969] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1434.111831] Lustre: Skipped 2 previous similar messages [ 1434.117628] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1434.121349] Lustre: Skipped 6 previous similar messages [ 1437.130193] LustreError: 64175:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1437.142920] LustreError: 64175:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1437.237654] Lustre: server umount lustre-OST0000 complete [ 1441.252916] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1441.255783] LustreError: Skipped 1 previous similar message [ 1444.834204] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1444.846129] Lustre: Skipped 2 previous similar messages [ 1446.558664] Lustre: server umount lustre-MDT0000 complete [ 1449.957421] LustreError: 61697:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1449.983846] LustreError: 61697:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1450.091257] LustreError: 61678:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763338118 with bad export cookie 4247505137009353859 [ 1450.097109] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1450.104113] LustreError: 61678:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1450.377217] Lustre: server umount lustre-MDT0001 complete [ 1460.199923] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 1463.225627] Key type lgssc unregistered [ 1463.492614] LNet: 65113:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1463.508953] LNetError: 65113:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1463.531177] LNet: Removed LNI 192.168.201.116@tcp [ 1464.399666] Key type .llcrypt unregistered [ 1464.406234] Key type ._llcrypt unregistered [ 1487.574059] Key type ._llcrypt registered [ 1487.575302] Key type .llcrypt registered [ 1487.666259] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 1488.466387] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1488.489541] alg: No test for adler32 (adler32-zlib) [ 1489.524428] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 1489.715369] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 1491.408442] Key type lgssc registered [ 1492.342488] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1499.162433] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1499.184045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1500.725321] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1504.723304] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1510.887467] LustreError: 66509:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1510.912707] LustreError: 66509:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1511.833033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1512.233160] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1515.359637] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1520.397948] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1524.088370] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1530.241638] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1530.909273] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1536.224810] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1541.847820] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1552.908979] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1561.575658] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1561.576620] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1561.607269] Lustre: Skipped 1 previous similar message [ 1561.609169] Lustre: Skipped 2 previous similar messages [ 1566.690310] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1566.710883] Lustre: Skipped 3 previous similar messages [ 1571.809369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1575.391282] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1575.529399] LustreError: 69220:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1575.669217] Lustre: server umount lustre-MDT0000 complete [ 1576.935236] LustreError: 66505:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1576.995427] LustreError: 66505:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1580.632839] LustreError: 66490:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763338249 with bad export cookie 8613201796228402921 [ 1580.649658] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1580.654856] LustreError: 66490:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1580.785302] LustreError: 69424:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1580.795437] LustreError: 69424:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1581.132623] Lustre: server umount lustre-MDT0001 complete [ 1595.920580] LustreError: 69625:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1595.922672] LustreError: 69625:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1596.062914] Lustre: server umount lustre-OST0000 complete [ 1604.847140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1609.150992] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1613.027092] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1616.353559] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1622.589506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1622.807385] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1622.829673] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1623.064972] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1626.403244] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1634.824639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1634.945840] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1634.977163] Lustre: 71076:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1635.401117] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1639.376702] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1644.996916] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1647.984867] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1653.616090] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1653.836316] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1654.120440] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1659.143267] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1664.257229] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1665.831638] Lustre: 73320:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1667.828784] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1675.232482] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1675.242539] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1675.257841] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1675.260734] Lustre: Skipped 2 previous similar messages [ 1675.747772] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1678.654084] LustreError: 73578:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1678.656746] LustreError: 73578:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1678.757930] Lustre: server umount lustre-OST0000 complete [ 1685.985824] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1685.992637] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1686.007499] Lustre: Skipped 2 previous similar messages [ 1686.021633] Lustre: Skipped 2 previous similar messages [ 1688.842505] LustreError: 73781:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1688.856193] LustreError: 73781:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1688.984214] Lustre: server umount lustre-MDT0000 complete [ 1691.106688] LustreError: 71909:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1691.120510] LustreError: 71909:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1692.188322] LustreError: 71808:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763338360 with bad export cookie 8613201796228403768 [ 1692.197347] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1692.209103] LustreError: 71808:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1692.474724] Lustre: server umount lustre-MDT0001 complete [ 1702.564211] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 1704.790379] Key type lgssc unregistered [ 1705.127184] LNet: 74516:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1705.136134] LNetError: 74516:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1705.156596] LNet: Removed LNI 192.168.201.116@tcp [ 1705.845733] Key type .llcrypt unregistered [ 1705.848923] Key type ._llcrypt unregistered [ 1728.217255] Key type ._llcrypt registered [ 1728.224161] Key type .llcrypt registered [ 1728.339758] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 1729.717575] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1729.736974] alg: No test for adler32 (adler32-zlib) [ 1730.738447] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 1731.003911] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 1732.727433] Key type lgssc registered [ 1733.929395] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1741.313456] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1741.336335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1742.761835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1746.177692] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1753.062364] LustreError: 75910:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1753.082373] LustreError: 75910:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1753.903424] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1754.428943] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1758.252588] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1763.123609] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1766.682677] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1773.248598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1773.798829] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1778.990387] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1784.501849] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1795.868182] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1796.967649] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1798.074506] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1800.146712] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1800.150388] Lustre: Skipped 1 previous similar message [ 1804.259560] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1804.288315] Lustre: Skipped 3 previous similar messages [ 1812.894272] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1812.903561] Lustre: Skipped 8 previous similar messages [ 1828.901501] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1828.905722] Lustre: Skipped 17 previous similar messages [ 1861.164855] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1861.170086] Lustre: Skipped 37 previous similar messages [ 1886.134074] Lustre: 83676:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 1888.534824] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 19:15:54 (1763338554) [ 1890.115720] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 1892.093090] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 19:15:57 (1763338557) [ 1896.932954] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1896.945505] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1896.978516] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1901.029074] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1901.030411] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1901.035251] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1902.343479] LustreError: 83972:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1902.424208] Lustre: server umount lustre-OST0000 complete [ 1907.680219] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1907.685982] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1907.707320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1907.713218] Lustre: Skipped 1 previous similar message [ 1911.264936] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1911.265194] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1911.287951] Lustre: Skipped 1 previous similar message [ 1912.509123] LustreError: 84173:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1912.516994] LustreError: 84173:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1912.609237] Lustre: server umount lustre-MDT0000 complete [ 1916.010649] LustreError: 78515:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763338584 with bad export cookie 7361880750538889188 [ 1916.018482] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1916.164995] LustreError: 84375:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1916.171507] LustreError: 84375:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1916.342223] Lustre: server umount lustre-MDT0001 complete [ 1925.452067] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 1928.100982] Key type lgssc unregistered [ 1928.389645] LNet: 84907:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1928.403903] LNetError: 84907:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1928.414587] LNet: Removed LNI 192.168.201.116@tcp [ 1929.101887] Key type .llcrypt unregistered [ 1929.104050] Key type ._llcrypt unregistered [ 1931.901243] Key type ._llcrypt registered [ 1931.902979] Key type .llcrypt registered [ 1931.992497] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 1935.595732] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules [ 1936.520645] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1936.643256] alg: No test for adler32 (adler32-zlib) [ 1937.686493] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 1937.894222] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 1939.551704] Key type lgssc registered [ 1940.652353] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1941.920992] LustreError: 86072:0:(lu_object.c:1438:lu_context_key_register()) cfs_race id 60d sleeping [ 1958.361752] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 1964.298768] LustreError: 86072:0:(lu_object.c:1438:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 1966.238549] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1966.268461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1967.709823] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1971.381358] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1972.717120] LustreError: 87078:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1972.741280] LustreError: 87078:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1977.827412] LustreError: 87078:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1978.997866] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 19:17:24 (1763338644) [ 1982.945138] LustreError: 87078:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1983.546636] LustreError: 87109:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1983.551474] LustreError: 87109:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1983.560949] LustreError: 87109:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 16, retries 0, failed: rc = -5 [ 1983.752562] LustreError: 87883:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1984.003457] Lustre: server umount lustre-MDT0000 complete [ 1996.745816] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 1999.456039] Key type lgssc unregistered [ 1999.798957] LNet: 88569:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1999.812419] LNetError: 88569:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1999.849780] LNet: Removed LNI 192.168.201.116@tcp [ 2000.733212] Key type .llcrypt unregistered [ 2000.734872] Key type ._llcrypt unregistered [ 2017.864028] Key type ._llcrypt registered [ 2017.865744] Key type .llcrypt registered [ 2017.971557] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 2018.810297] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2018.830780] alg: No test for adler32 (adler32-zlib) [ 2020.023455] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 2020.264149] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 2021.919342] Key type lgssc registered [ 2022.938230] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2029.895185] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2029.918144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2031.369260] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2035.952664] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2040.820622] LustreError: 89768:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2040.842932] LustreError: 89768:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2043.846388] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2044.481410] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2048.138540] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2054.059986] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2058.017433] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2064.017282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2064.466443] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2068.931751] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2073.792687] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2117.280323] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 19:19:43 (1763338783) [ 2119.236387] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2121.189771] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 19:19:46 (1763338786) [ 2126.305936] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2126.316703] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2126.327667] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2127.330525] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2127.334192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2127.345366] Lustre: Skipped 1 previous similar message [ 2127.353901] Lustre: Skipped 2 previous similar messages [ 2130.963935] LustreError: 92678:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2131.095105] Lustre: server umount lustre-MDT0000 complete [ 2132.448535] LustreError: 89767:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2132.487735] LustreError: 89767:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2135.168040] LustreError: 89748:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763338803 with bad export cookie 10237923729638777068 [ 2135.169661] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2135.181990] LustreError: 89748:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2135.375346] LustreError: 92880:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2135.393372] LustreError: 92880:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2135.648838] Lustre: server umount lustre-MDT0001 complete [ 2149.856319] LustreError: 93083:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2149.861947] LustreError: 93083:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2150.209069] Lustre: server umount lustre-OST0000 complete [ 2161.856884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2162.253520] LustreError: 93713:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2162.446420] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2165.753757] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2167.777675] LustreError: 93713:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2171.874379] LustreError: 93713:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2172.882091] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2173.540390] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2177.153480] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2183.032211] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2186.433467] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2192.593954] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2198.953111] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2199.040624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2199.301937] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2199.311660] LustreError: 93702:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2199.344912] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2199.363158] LustreError: 95575:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -98 [ 2199.374117] LustreError: 95575:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 2199.378585] LustreError: 95575:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 2199.443064] Lustre: server umount lustre-OST0000 complete [ 2199.445428] LustreError: 95575:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2206.539808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2206.805280] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2206.814178] LustreError: 93702:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2206.839052] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2206.848413] LustreError: 96041:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -98 [ 2206.854801] LustreError: 96041:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 2206.863694] LustreError: 96041:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 2206.920318] Lustre: server umount lustre-OST0000 complete [ 2206.923104] LustreError: 96041:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2209.489477] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2214.699323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2214.829996] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2214.833044] Lustre: Found index 0 for lustre-OST0000, updating log [ 2214.840324] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2214.905389] Lustre: lustre-OST0000: new disk, initializing [ 2214.914413] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2215.154870] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2220.114918] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2223.464520] Lustre: Failing over lustre-OST0000 [ 2223.517199] LustreError: 97128:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2223.525735] LustreError: 97128:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2223.612824] Lustre: server umount lustre-OST0000 complete [ 2226.625188] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2226.627798] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2226.638119] Lustre: Skipped 1 previous similar message [ 2231.780836] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2231.785706] Lustre: Skipped 2 previous similar messages [ 2232.830517] LustreError: 97328:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2232.841734] LustreError: 97328:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2232.934095] Lustre: server umount lustre-MDT0000 complete [ 2236.475689] LustreError: 93694:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763338904 with bad export cookie 10237923729638779812 [ 2236.481591] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2236.487766] LustreError: 93694:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2246.125138] LustreError: 97531:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2246.138274] LustreError: 97531:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2246.313833] Lustre: server umount lustre-MDT0001 complete [ 2254.269769] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 19:22:00 (1763338920) [ 2268.936877] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 2278.468091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2278.889369] LustreError: 98901:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2279.019598] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2282.767243] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2291.486401] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2295.931917] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2298.461123] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2304.814330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2305.478983] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2310.756566] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2316.708758] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2316.741661] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2316.744844] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2319.291927] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2319.367295] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2319.431469] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2319.558863] Lustre: lustre-OST0001: new disk, initializing [ 2319.570936] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2319.857137] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2319.864943] Lustre: Skipped 2 previous similar messages [ 2325.306254] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2329.218417] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2329.227540] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2329.328267] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2332.120557] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2337.852714] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2348.238228] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.16@tcp (stopping) [ 2348.514267] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2348.523666] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2352.309642] LustreError: 102141:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 2352.317825] LustreError: 102141:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2352.376763] Lustre: server umount lustre-MDT0001 complete [ 2353.343589] LustreError: 98897:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2353.353225] LustreError: 98897:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2354.434458] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2359.899300] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2360.010610] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2360.013188] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2360.040330] Lustre: 98890:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2361.724487] LustreError: 102753:0:(obd_mount.c:188:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2361.731246] LustreError: 102753:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2361.741020] LustreError: 98892:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2361.824772] LustreError: 102760:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.201.116@tcp: cfg command failed: rc = -17 [ 2361.835423] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2361.835423] [ 2364.125828] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2365.929863] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 2372.094474] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID ========================================================== 19:23:58 (1763339038) [ 2388.058944] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 2404.116375] Lustre: 104755:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2426.320690] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2430.666390] Lustre: 105889:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2519.527859] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2519.528901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2519.542475] Lustre: Skipped 3 previous similar messages [ 2519.552833] Lustre: Skipped 4 previous similar messages [ 2524.128870] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2525.449629] LustreError: 106345:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 2525.454328] LustreError: 106345:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2527.566956] Lustre: server umount lustre-MDT0000 complete [ 2529.765372] LustreError: 99626:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2529.774689] LustreError: 99626:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 2530.688561] LustreError: 98880:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763339199 with bad export cookie 10237923729638780813 [ 2530.691182] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2530.701430] LustreError: 98880:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2530.989531] Lustre: server umount lustre-MDT0001 complete [ 2545.223286] Lustre: server umount lustre-OST0000 complete [ 2558.531681] Lustre: server umount lustre-OST0001 complete [ 2563.237447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2570.208155] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2570.964964] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2577.704820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2585.641686] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2586.555643] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2601.961227] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 2609.718411] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2609.730056] Lustre: lustre-MDT0000: reset Object Index mappings [ 2610.086477] LustreError: 109829:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2610.171811] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2610.176404] Lustre: Skipped 1 previous similar message [ 2613.122638] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2619.364043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2619.379837] Lustre: lustre-MDT0001: reset Object Index mappings [ 2622.523044] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2624.669495] Lustre: 110945:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2629.378611] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2630.891472] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:231 to 0x280000400:257) [ 2633.142689] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2636.015785] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:350 to 0x280000401:386) [ 2639.051820] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2640.829705] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 2640.870020] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 2644.036904] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2650.192208] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2652.845172] Lustre: 112796:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2697.204972] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 19:29:23 (1763339363) [ 2726.885701] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2726.887787] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2726.893951] Lustre: Skipped 5 previous similar messages [ 2726.900199] Lustre: Skipped 6 previous similar messages [ 2729.254460] LustreError: 113147:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2729.257563] LustreError: 113147:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 2729.344471] Lustre: server umount lustre-MDT0000 complete [ 2732.806124] LustreError: 113148:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763339401 with bad export cookie 10237923729638827671 [ 2732.807042] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2732.811710] LustreError: 113148:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2742.243527] LustreError: 109823:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2742.255582] LustreError: 109823:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 13 previous similar messages [ 2747.359559] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2747.608823] Lustre: server umount lustre-MDT0001 complete [ 2765.279501] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2765.382443] Lustre: server umount lustre-OST0000 complete [ 2774.613404] Lustre: server umount lustre-OST0001 complete [ 2780.465412] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 2790.039444] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 2800.074332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2807.408987] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2812.954390] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2818.319649] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2823.766487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2823.821596] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2824.014260] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2824.048933] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2824.134449] Lustre: lustre-MDT0000: new disk, initializing [ 2824.211702] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2824.217177] Lustre: Skipped 3 previous similar messages [ 2824.236367] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2827.679599] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2833.540437] Lustre: server umount lustre-MDT0000 complete [ 2835.470722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2840.427944] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2840.495904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2840.740684] Lustre: Found index 0 for lustre-MDT0000, updating log [ 2840.751396] Lustre: 117030:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2840.760964] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 2844.269840] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2848.458145] Lustre: server umount lustre-MDT0000 complete [ 2856.664600] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 19:32:02 (1763339522) [ 2858.066651] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 2859.903781] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 19:32:05 (1763339525) [ 2875.143286] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 2883.540745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2886.953595] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2894.683054] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2894.746745] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2894.826654] Lustre: 119047:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2894.881641] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2894.887627] Lustre: Skipped 1 previous similar message [ 2894.943940] Lustre: lustre-MDT0001: new disk, initializing [ 2895.070550] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2895.083616] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2898.625883] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2902.926964] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 2908.905646] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2908.967522] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2909.179423] Lustre: lustre-OST0000: new disk, initializing [ 2909.186391] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2910.504502] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2910.517706] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2910.661436] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2914.537882] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2924.120841] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2924.183240] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2924.314126] Lustre: lustre-OST0001: new disk, initializing [ 2924.319884] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2926.087057] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2926.092431] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2926.222054] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2929.843364] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2938.967891] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2942.823527] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3324.271888] Lustre: 127857:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3324.277447] Lustre: 127857:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3324.291062] Lustre: 127857:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3324.295190] Lustre: 127857:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3324.299293] Lustre: 127857:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3324.311521] Lustre: 127857:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3582.350049] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 19:44:08 (1763340248) [ 3596.772897] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3596.788788] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3596.808918] Lustre: Skipped 2 previous similar messages [ 3596.814375] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3596.819524] Lustre: Skipped 7 previous similar messages [ 3601.839759] LustreError: 128107:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 3601.845234] LustreError: 128107:0:(obd_class.h:479:obd_check_dev()) Skipped 33 previous similar messages [ 3602.032333] Lustre: server umount lustre-MDT0000 complete [ 3602.916560] LustreError: 122243:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3602.924043] LustreError: 122243:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3604.618458] LustreError: 119040:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763340273 with bad export cookie 10237923729638891980 [ 3604.624659] LustreError: 119040:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3604.626952] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3608.037248] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3608.042601] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3608.047550] Lustre: Skipped 4 previous similar messages [ 3608.053137] Lustre: Skipped 5 previous similar messages [ 3610.949567] Lustre: server umount lustre-MDT0001 complete [ 3619.903095] LustreError: 128512:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3619.906197] LustreError: 128512:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 3619.951435] Lustre: server umount lustre-OST0000 complete [ 3628.471996] Lustre: server umount lustre-OST0001 complete [ 3633.478438] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 3642.333308] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 3650.663137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3656.337734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3662.113382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3667.895639] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3673.063106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3673.111473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3673.241265] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3673.271995] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3673.319822] Lustre: lustre-MDT0000: new disk, initializing [ 3673.381246] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3673.384456] Lustre: Skipped 5 previous similar messages [ 3673.398194] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3676.229630] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3684.615811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3684.679198] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3684.743915] Lustre: 130899:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3684.795635] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3684.798899] Lustre: Skipped 1 previous similar message [ 3684.849276] Lustre: lustre-MDT0001: new disk, initializing [ 3684.958901] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3684.985261] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3688.053283] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3693.337329] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3695.572483] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3700.127322] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3700.174411] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3700.328976] Lustre: lustre-OST0000: new disk, initializing [ 3700.332680] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3701.493742] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3701.500320] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3701.579466] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3704.461589] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3709.760392] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3717.264229] Lustre: lustre-MDD0000: changelog on [ 3719.174403] Lustre: lustre-MDD0001: changelog on [ 5569.492345] Lustre: lustre-MDD0001: changelog off [ 5571.294103] Lustre: 130908:0:(llog_cat.c:971:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 5572.079238] Lustre: lustre-MDD0000: changelog off [ 5580.768687] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5580.768862] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5580.779251] LustreError: Skipped 1 previous similar message [ 5580.795156] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5580.800137] Lustre: Skipped 3 previous similar messages [ 5585.891658] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5585.906502] Lustre: Skipped 1 previous similar message [ 5591.017380] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5591.023432] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5591.056407] Lustre: Skipped 1 previous similar message [ 5591.091659] LustreError: 140708:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5591.094452] LustreError: 140708:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 5591.237453] Lustre: server umount lustre-OST0000 complete [ 5596.136460] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5596.142565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5596.154947] Lustre: Skipped 2 previous similar messages [ 5601.267854] LustreError: 140910:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5601.275820] LustreError: 140910:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5601.468637] Lustre: server umount lustre-MDT0000 complete [ 5604.804820] LustreError: 130891:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763342273 with bad export cookie 10237923729639707053 [ 5604.815186] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5604.823220] LustreError: 130891:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5605.104973] Lustre: server umount lustre-MDT0001 complete [ 5615.156228] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 5617.495912] Key type lgssc unregistered [ 5617.729132] LNet: 141643:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5617.738352] LNetError: 141643:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5617.751808] LNet: Removed LNI 192.168.201.116@tcp [ 5618.373685] Key type .llcrypt unregistered [ 5618.375569] Key type ._llcrypt unregistered [ 5635.465547] Key type ._llcrypt registered [ 5635.468838] Key type .llcrypt registered [ 5635.554524] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 5651.463124] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 5652.887939] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5653.009482] alg: No test for adler32 (adler32-zlib) [ 5654.014037] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 5654.205804] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 5655.871181] Key type lgssc registered [ 5657.355626] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5665.783336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5671.387956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5676.691148] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5682.036744] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5687.736751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5687.773478] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5687.786131] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5688.989150] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5689.024914] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5689.094868] Lustre: lustre-MDT0000: new disk, initializing [ 5689.185324] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5689.206625] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5692.403884] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5700.913139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5700.980708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5701.069219] Lustre: 144818:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5701.139376] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5701.145946] Lustre: Skipped 1 previous similar message [ 5701.234697] Lustre: lustre-MDT0001: new disk, initializing [ 5701.314574] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5701.348738] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5701.359384] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5704.345325] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5710.937631] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5714.559434] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5721.022500] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5721.123896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5721.381520] Lustre: lustre-OST0000: new disk, initializing [ 5721.397355] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5721.493950] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5726.539536] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5728.292419] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5728.303799] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5728.369649] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5732.885426] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5738.066783] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_module obdecho/obdecho [ 5740.018074] LustreError: 147439:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 5740.026388] LustreError: 147439:0:(obd_config.c:783:class_setup()) setup ec failed (-22) [ 5741.508546] LustreError: 147535:0:(echo_client.c:1880:echo_md_handler()) Can not resolve the path : rc = -2 [ 5753.827275] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5753.832507] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5753.845362] Lustre: Skipped 1 previous similar message [ 5753.861212] Lustre: Skipped 2 previous similar messages [ 5758.946684] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5758.966672] Lustre: Skipped 2 previous similar messages [ 5759.491559] LustreError: 147876:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5759.674916] Lustre: server umount lustre-MDT0000 complete [ 5763.195212] LustreError: 144811:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763342431 with bad export cookie 5862426950230646907 [ 5763.213052] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5763.408342] LustreError: 148078:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5763.456270] LustreError: 148078:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5763.647940] Lustre: server umount lustre-MDT0001 complete [ 5777.423095] LustreError: 148279:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5777.432044] LustreError: 148279:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5777.558421] Lustre: server umount lustre-OST0000 complete [ 5795.441911] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 5804.963182] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 5812.933689] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5818.923795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5824.129595] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5829.209413] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5834.999283] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5835.046355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5835.242668] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5835.273885] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5835.324366] Lustre: lustre-MDT0000: new disk, initializing [ 5835.396725] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5835.410604] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5838.840135] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5848.140634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5848.213247] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5848.286762] Lustre: 151229:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5848.335489] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5848.338464] Lustre: Skipped 1 previous similar message [ 5848.384927] Lustre: lustre-MDT0001: new disk, initializing [ 5848.462027] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5848.498109] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5848.508101] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5851.822853] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5858.439608] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5861.689764] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5866.479212] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5866.544249] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5866.733285] Lustre: lustre-OST0000: new disk, initializing [ 5866.736454] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5866.795728] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5871.035480] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5875.260475] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5875.277283] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5875.398522] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5878.002538] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5881.545449] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5881.725588] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5885.021672] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5885.186510] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5891.047496] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5891.047523] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5891.050876] LustreError: Skipped 1 previous similar message [ 5891.058109] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5891.060711] Lustre: Skipped 2 previous similar messages [ 5891.071189] Lustre: Skipped 1 previous similar message [ 5896.167875] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5896.180138] Lustre: Skipped 1 previous similar message [ 5901.279245] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5901.280289] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5901.294084] Lustre: Skipped 1 previous similar message [ 5901.324377] LustreError: 154165:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5901.333094] LustreError: 154165:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5901.437906] Lustre: server umount lustre-OST0000 complete [ 5904.865967] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5904.878756] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5910.746813] LustreError: 154368:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5910.755091] LustreError: 154368:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5910.806594] Lustre: server umount lustre-MDT0000 complete [ 5913.537130] LustreError: 151220:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763342581 with bad export cookie 5862426950230647859 [ 5913.542880] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5913.549744] LustreError: 151220:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 5913.820779] Lustre: server umount lustre-MDT0001 complete [ 5917.079335] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 20:23:03 (1763342583) [ 5918.527323] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 5920.140279] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 20:23:06 (1763342586) [ 5926.579773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5926.886946] LustreError: 155240:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5926.902990] LustreError: 155240:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5926.954347] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5930.277625] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5932.004100] LustreError: 155240:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5935.010457] LustreError: 155240:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5937.057840] LustreError: 155240:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5937.267275] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5940.562172] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5945.066572] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5948.483495] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5955.196778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5960.109885] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5964.490169] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5981.106823] LustreError: 157509:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5981.112776] LustreError: 157509:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 5981.228403] Lustre: server umount lustre-MDT0000 complete [ 5982.179630] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5982.183136] LustreError: 155239:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5982.191303] Lustre: Skipped 2 previous similar messages [ 5982.215900] LustreError: 155239:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5984.745903] LustreError: 155221:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763342653 with bad export cookie 5862426950230648748 [ 5984.757080] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5984.762087] LustreError: 155221:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5985.475236] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.16@tcp (stopping) [ 5985.478804] Lustre: Skipped 3 previous similar messages [ 5987.297459] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5987.314366] Lustre: Skipped 1 previous similar message [ 5990.581881] LustreError: 155236:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5990.601923] LustreError: 155236:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5999.071158] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5999.166078] LustreError: 157711:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5999.173147] LustreError: 157711:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5999.367661] Lustre: server umount lustre-MDT0001 complete [ 6005.052070] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6015.770933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6015.938624] LustreError: 158773:0:(llog.c:1616:llog_backup()) MGC192.168.201.116@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6015.946739] Lustre: 158773:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.201.116@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6026.212276] LustreError: 158781:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6026.225340] LustreError: 158781:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6059.619384] LustreError: 158780:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6059.652378] LustreError: 158780:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 30 previous similar messages [ 6076.895312] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6076.904131] Lustre: 158773:0:(mgc_request.c:1910:mgc_process_log()) MGC192.168.201.116@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6105.055397] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6105.062478] Lustre: 155232:0:(mgc_request.c:1910:mgc_process_log()) MGC192.168.201.116@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6117.898488] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6117.901436] Lustre: Skipped 2 previous similar messages [ 6121.507624] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6122.983051] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6125.553361] LustreError: 155232:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 6125.753606] LustreError: 158779:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6125.763631] LustreError: 158779:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 41 previous similar messages [ 6130.132040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6145.507312] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 6158.393092] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6160.364969] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6163.668973] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6165.304424] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6168.782603] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6170.408582] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6187.069852] LustreError: 160284:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6187.078519] LustreError: 160284:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6187.198110] Lustre: server umount lustre-MDT0000 complete [ 6190.416026] LustreError: 157712:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763342858 with bad export cookie 5862426950230655265 [ 6190.421411] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6190.424744] LustreError: 157712:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6190.721801] Lustre: server umount lustre-MDT0001 complete [ 6214.623697] Lustre: server umount lustre-OST0000 complete [ 6236.344251] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 6245.407061] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 6253.492508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6259.816362] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6265.906544] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6272.212941] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6279.145447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6279.250105] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6279.424526] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6279.455563] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6279.505345] Lustre: lustre-MDT0000: new disk, initializing [ 6279.576364] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6279.587405] Lustre: Skipped 1 previous similar message [ 6279.607835] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6282.862024] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6293.915690] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6293.965110] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6294.031096] Lustre: 163635:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6294.118505] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6294.129697] Lustre: Skipped 6 previous similar messages [ 6294.281704] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6300.060255] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6307.618529] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6311.876250] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6318.465944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6318.537205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6318.709649] Lustre: lustre-OST0000: new disk, initializing [ 6318.711833] Lustre: Skipped 1 previous similar message [ 6318.714667] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6318.717764] Lustre: Skipped 2 previous similar messages [ 6324.440045] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6328.878256] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6328.885860] Lustre: Skipped 1 previous similar message [ 6328.888465] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6328.968895] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6331.335460] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6335.506907] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6335.737409] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6339.251969] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6339.466359] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6344.164512] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6344.169439] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6344.182672] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6344.190985] Lustre: Skipped 1 previous similar message [ 6347.526659] LustreError: 166576:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6347.531650] LustreError: 166576:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 6347.661467] Lustre: server umount lustre-OST0000 complete [ 6357.383706] Lustre: server umount lustre-MDT0000 complete [ 6359.526557] LustreError: 165686:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6359.548764] LustreError: 165686:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 27 previous similar messages [ 6360.421270] LustreError: 163627:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763343028 with bad export cookie 5862426950230672058 [ 6360.426439] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6360.432402] LustreError: 163627:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6360.728562] Lustre: server umount lustre-MDT0001 complete [ 6367.710876] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 20:30:34 (1763343034) [ 6373.207184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6376.638730] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6382.607205] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6385.701514] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6390.443684] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6394.370893] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6400.633270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6405.678921] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6410.539816] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6429.536710] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 20:31:35 (1763343095) [ 6433.760258] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6433.764800] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6433.771590] Lustre: Skipped 3 previous similar messages [ 6433.775410] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6433.779019] Lustre: Skipped 2 previous similar messages [ 6438.657282] Lustre: server umount lustre-OST0000 complete [ 6444.512233] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6444.519837] LustreError: Skipped 1 previous similar message [ 6447.955379] Lustre: server umount lustre-MDT0000 complete [ 6450.594267] LustreError: 167633:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763343119 with bad export cookie 5862426950230672940 [ 6450.598707] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6450.605260] LustreError: 167633:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6450.895753] Lustre: server umount lustre-MDT0001 complete [ 6457.004812] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 6458.941886] Key type lgssc unregistered [ 6459.171260] LNet: 171049:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6459.182100] LNetError: 171049:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6459.199457] LNet: Removed LNI 192.168.201.116@tcp [ 6459.776277] Key type .llcrypt unregistered [ 6459.779644] Key type ._llcrypt unregistered [ 6473.973442] Key type ._llcrypt registered [ 6473.975418] Key type .llcrypt registered [ 6474.071965] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 6474.712502] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6474.751713] alg: No test for adler32 (adler32-zlib) [ 6475.819647] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 6475.984445] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 6477.623189] Key type lgssc registered [ 6478.530400] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6484.107473] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6484.120853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6499.617491] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6504.742656] LustreError: 172250:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.116@tcp: failed processing log, type 4: rc = -110 [ 6509.919523] LustreError: 172250:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6509.938552] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 6509.944147] LustreError: MGC192.168.201.116@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 6509.959517] LustreError: 172250:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-OST0000: -22 [ 6509.973867] LustreError: 172250:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -22 [ 6509.979628] LustreError: 172250:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 6509.989167] LustreError: 172250:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 6510.081163] Lustre: server umount lustre-OST0000 complete [ 6510.092828] LustreError: 172250:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6517.167368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6518.524385] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6521.423949] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6527.971337] LustreError: 172752:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6528.008222] LustreError: 172752:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6530.535483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6530.920775] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6534.995862] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6541.040784] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6544.635929] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6550.404897] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6550.639485] LustreError: 174345:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6550.656690] Lustre: 174345:0:(mgc_request_server.c:712:mgc_llog_local_copy()) MGC192.168.201.116@tcp: invalid local config log lustre-OST0000: rc = -22 [ 6550.806456] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6555.470619] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6560.198787] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6566.369198] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6566.375553] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6566.378476] Lustre: Skipped 1 previous similar message [ 6568.048310] LustreError: 174997:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6568.051417] LustreError: 174997:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 6568.184659] Lustre: server umount lustre-MDT0000 complete [ 6571.490027] LustreError: 172752:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6571.497433] LustreError: 172752:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6571.615829] LustreError: 174998:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763343240 with bad export cookie 5517141591025342086 [ 6571.626542] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6571.637855] LustreError: 174998:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6571.837424] LustreError: 175199:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6571.857278] LustreError: 175199:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6571.987145] Lustre: server umount lustre-MDT0001 complete [ 6585.597251] LustreError: 175400:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6585.599685] LustreError: 175400:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6585.739709] Lustre: server umount lustre-OST0000 complete [ 6593.054447] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 6600.691296] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 6608.664674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6614.377102] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6619.508050] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6625.443710] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6631.681535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6631.756549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6631.995222] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6632.048561] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6632.121994] Lustre: lustre-MDT0000: new disk, initializing [ 6632.241873] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6632.259209] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6636.256364] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6645.181490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6645.263728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6645.334727] Lustre: 177738:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6645.397106] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6645.408485] Lustre: Skipped 1 previous similar message [ 6645.482271] Lustre: lustre-MDT0001: new disk, initializing [ 6645.572585] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6645.607210] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6645.616316] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6648.781860] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6655.390533] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6658.847161] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6664.274841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6664.355649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6664.530774] Lustre: lustre-OST0000: new disk, initializing [ 6664.537930] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6664.629104] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6669.714440] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6674.491665] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6674.500676] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6674.584582] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6676.326780] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6679.483557] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6679.725128] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6683.635801] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6683.832504] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6690.276609] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6690.277902] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6690.284687] Lustre: Skipped 3 previous similar messages [ 6690.284736] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6690.288449] LustreError: Skipped 1 previous similar message [ 6691.579134] LustreError: 180676:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6691.586687] LustreError: 180676:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6691.708813] Lustre: server umount lustre-OST0000 complete [ 6695.392658] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6695.399499] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6695.402862] Lustre: Skipped 1 previous similar message [ 6695.415357] Lustre: Skipped 2 previous similar messages [ 6700.516395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6700.526112] Lustre: Skipped 1 previous similar message [ 6700.787792] LustreError: 180879:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6700.791602] LustreError: 180879:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6700.890481] Lustre: server umount lustre-MDT0000 complete [ 6703.670973] LustreError: 177730:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763343372 with bad export cookie 5517141591025342800 [ 6703.676147] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6703.678103] LustreError: 177730:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6703.936967] Lustre: server umount lustre-MDT0001 complete [ 6711.186119] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 20:36:17 (1763343377) [ 6726.796926] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 6729.288887] Key type lgssc unregistered [ 6729.568576] LNet: 182264:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6729.575868] LNetError: 182264:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6729.587775] LNet: Removed LNI 192.168.201.116@tcp [ 6730.340119] Key type .llcrypt unregistered [ 6730.341861] Key type ._llcrypt unregistered [ 6744.211558] Key type ._llcrypt registered [ 6744.215964] Key type .llcrypt registered [ 6744.324467] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 6745.361975] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6745.402179] alg: No test for adler32 (adler32-zlib) [ 6746.481774] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 6746.695588] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 6748.447174] Key type lgssc registered [ 6749.451532] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6756.030827] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6756.059644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6757.463549] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6760.282455] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6766.261844] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6766.518896] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6769.210365] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6773.785272] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6777.207705] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6783.702556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6783.844091] Lustre: *** cfs_fail_loc=131e, val=0*** [ 6783.848370] LustreError: 185087:0:(llog.c:1657:llog_backup()) MGC192.168.201.116@tcp: failed to backup log lustre-OST0000: rc = -28 [ 6783.854851] Lustre: 185087:0:(mgc_request_server.c:709:mgc_llog_local_copy()) MGC192.168.201.116@tcp: can't backup local config lustre-OST0000: rc = -28 [ 6783.950245] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6788.571332] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6792.606880] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6796.262274] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6796.263145] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6796.278548] Lustre: Skipped 2 previous similar messages [ 6796.304868] Lustre: Skipped 1 previous similar message [ 6800.858919] LustreError: 185739:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6800.945524] Lustre: server umount lustre-MDT0000 complete [ 6801.378607] LustreError: 184157:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6801.399431] LustreError: 184157:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6804.007785] LustreError: 185740:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763343472 with bad export cookie 9245164389958232554 [ 6804.013599] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6804.122663] LustreError: 185942:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6804.125112] LustreError: 185942:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6804.253758] Lustre: server umount lustre-MDT0001 complete [ 6817.533880] LustreError: 186142:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6817.546479] LustreError: 186142:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6817.701625] Lustre: server umount lustre-OST0000 complete [ 6825.828127] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 6833.603514] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 6842.089392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6847.777493] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6852.714533] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6859.160840] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6865.346454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6865.405524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6865.559969] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6865.585189] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6865.630337] Lustre: lustre-MDT0000: new disk, initializing [ 6865.689353] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6865.706959] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6868.431925] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6876.082333] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6876.132296] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6876.182569] Lustre: 188481:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6876.212527] Lustre: *** cfs_fail_loc=131e, val=0*** [ 6876.214339] Lustre: Skipped 1 previous similar message [ 6876.215784] LustreError: 189248:0:(llog.c:1657:llog_backup()) MGC192.168.201.116@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 6876.217699] LustreError: 189248:0:(llog.c:1657:llog_backup()) Skipped 1 previous similar message [ 6876.220034] Lustre: 189248:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.201.116@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 6876.233448] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6876.236385] Lustre: Skipped 1 previous similar message [ 6876.276058] Lustre: lustre-MDT0001: new disk, initializing [ 6876.343145] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6876.365561] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6876.369944] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6878.668316] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6885.071583] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6888.839768] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6895.578353] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6895.655639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6895.854182] Lustre: *** cfs_fail_loc=131e, val=0*** [ 6895.856287] Lustre: Skipped 1 previous similar message [ 6895.859147] LustreError: 190203:0:(llog.c:1657:llog_backup()) MGC192.168.201.116@tcp: failed to backup log lustre-OST0000: rc = -28 [ 6895.863464] LustreError: 190203:0:(llog.c:1657:llog_backup()) Skipped 1 previous similar message [ 6895.866510] Lustre: 190203:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.201.116@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 6895.871469] Lustre: 190203:0:(mgc_request_server.c:732:mgc_llog_local_copy()) Skipped 1 previous similar message [ 6895.890157] Lustre: lustre-OST0000: new disk, initializing [ 6895.904718] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6895.983144] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6900.810842] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6905.424288] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6905.439062] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6905.561864] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6907.396355] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6910.719497] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6910.906065] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6914.759766] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6914.938328] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6921.189335] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6921.189556] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6921.196705] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6921.199409] LustreError: Skipped 1 previous similar message [ 6922.678116] LustreError: 191420:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6922.687726] LustreError: 191420:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6922.799966] Lustre: server umount lustre-OST0000 complete [ 6926.305981] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6926.311664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6926.320317] Lustre: Skipped 2 previous similar messages [ 6926.327850] Lustre: Skipped 1 previous similar message [ 6931.428642] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6931.435246] Lustre: Skipped 1 previous similar message [ 6932.110294] LustreError: 191622:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6932.113352] LustreError: 191622:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6932.192298] Lustre: server umount lustre-MDT0000 complete [ 6934.946698] LustreError: 188472:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763343603 with bad export cookie 9245164389958233268 [ 6934.967772] LustreError: 188472:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 6934.971693] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6935.231789] Lustre: server umount lustre-MDT0001 complete [ 6941.889857] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 20:40:08 (1763343608) [ 6955.639651] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 6965.541308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6965.921114] LustreError: 193189:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6965.991296] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6968.863428] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6971.360712] LustreError: 193189:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6974.430990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6978.098137] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6980.625400] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6986.575151] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6992.343721] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7000.715667] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7000.809480] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7001.006558] Lustre: lustre-OST0001: new disk, initializing [ 7001.012365] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7001.125200] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7001.132752] Lustre: Skipped 2 previous similar messages [ 7007.942874] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7007.954109] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7008.133938] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7008.537338] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7019.700724] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7029.160902] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7036.056249] Lustre: DEBUG MARKER: ADD OST3 [ 7040.105530] loop: module loaded [ 7040.181794] loop0: detected capacity change from 0 to 204800000 [ 7040.317078] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7040.336501] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7040.352884] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7040.491194] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7045.250182] loop0: detected capacity change from 0 to 204800000 [ 7046.833063] Lustre: DEBUG MARKER: STOP OST3 [ 7049.174407] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7049.238922] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7049.365402] Lustre: lustre-OST0003: new disk, initializing [ 7049.369321] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7050.836626] Lustre: Failing over lustre-OST0003 [ 7050.890671] LustreError: 197548:0:(obd_class.h:479:obd_check_dev()) Device 31 not setup [ 7050.896780] LustreError: 197548:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 7050.935763] Lustre: server umount lustre-OST0003 complete [ 7051.139732] LustreError: 195408:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7053.645891] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7055.572579] LustreError: 194645:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.201.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7055.582370] LustreError: 194645:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7061.705689] Lustre: DEBUG MARKER: START OST3 again [ 7065.795872] LustreError: 194646:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.201.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7065.808247] LustreError: 194646:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 7068.389879] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7068.608353] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 7068.615640] Lustre: Skipped 1 previous similar message [ 7068.623585] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 7070.643505] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 1 client reconnects [ 7070.667765] Lustre: lustre-OST0003: Denying connection for new client lustre-MDT0000-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:09 [ 7070.689727] LustreError: lustre-OST0003-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -16 [ 7070.701651] Lustre: lustre-OST0003: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 7073.232460] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7073.782513] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7073.789695] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7073.859185] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000401 [ 7093.964566] Lustre: Failing over lustre-OST0003 [ 7094.017108] LustreError: 199791:0:(obd_class.h:479:obd_check_dev()) Device 33 not setup [ 7094.022578] LustreError: 199791:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7094.072198] Lustre: server umount lustre-OST0003 complete [ 7094.259951] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7094.266579] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7094.278151] LustreError: 194644:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7094.292169] LustreError: Skipped 2 previous similar messages [ 7094.312872] LustreError: 194644:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7113.185096] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7113.191985] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7113.202122] Lustre: Skipped 1 previous similar message [ 7113.209328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7117.045830] Lustre: server umount lustre-MDT0000 complete [ 7120.074276] LustreError: 197375:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763343788 with bad export cookie 9245164389958234143 [ 7120.078211] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7120.087175] LustreError: 197375:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7124.962630] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7124.974926] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7124.976768] Lustre: Skipped 4 previous similar messages [ 7124.987831] Lustre: Skipped 5 previous similar messages [ 7126.331899] LustreError: 200265:0:(obd_class.h:479:obd_check_dev()) Device 29 not setup [ 7126.335374] LustreError: 200265:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 7126.437342] Lustre: server umount lustre-MDT0001 complete [ 7130.218237] Lustre: server umount lustre-OST0000 complete [ 7134.023264] Lustre: server umount lustre-OST0001 complete [ 7140.010378] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 7147.797675] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 7158.251929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7164.367331] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7169.256364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7174.490176] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7180.748455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7180.790804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7181.032718] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7181.091756] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7181.205476] Lustre: lustre-MDT0000: new disk, initializing [ 7181.331745] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7185.317507] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7195.461180] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7195.517327] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7195.616701] Lustre: 202853:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7195.905391] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7199.576963] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7206.284968] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7209.405316] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7214.420299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7214.474647] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7214.660871] Lustre: lustre-OST0000: new disk, initializing [ 7214.663501] Lustre: Skipped 1 previous similar message [ 7214.666457] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7214.669805] Lustre: Skipped 2 previous similar messages [ 7214.749435] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7214.754044] Lustre: Skipped 2 previous similar messages [ 7216.368948] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7216.385915] Lustre: Skipped 1 previous similar message [ 7216.391309] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7216.487687] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7219.945153] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7226.648094] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7230.331052] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7230.544376] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7234.454980] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7234.632848] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7237.095191] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7237.098282] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7237.099939] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7237.105012] Lustre: Skipped 1 previous similar message [ 7237.137754] Lustre: Skipped 3 previous similar messages [ 7242.246583] LustreError: 205794:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7242.256164] LustreError: 205794:0:(obd_class.h:479:obd_check_dev()) Skipped 19 previous similar messages [ 7242.375846] Lustre: server umount lustre-OST0000 complete [ 7247.336792] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7247.358133] LustreError: Skipped 1 previous similar message [ 7251.795753] Lustre: server umount lustre-MDT0000 complete [ 7252.448841] LustreError: 202865:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7252.463746] LustreError: 202865:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 18 previous similar messages [ 7255.315304] LustreError: 205997:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763343923 with bad export cookie 9245164389958236432 [ 7255.319369] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7255.329402] LustreError: 205997:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7255.661985] Lustre: server umount lustre-MDT0001 complete [ 7258.896873] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 20:45:24 (1763343924) [ 7274.534951] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 7285.157471] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 7296.188787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7301.066073] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7306.073681] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7311.269747] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7316.851501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7316.937779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7317.137601] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7317.162387] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7317.208929] Lustre: lustre-MDT0000: new disk, initializing [ 7317.305960] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7320.767416] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7330.228757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7330.312949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7330.441149] Lustre: 209044:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7330.757110] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7334.515937] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7340.184754] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7342.611483] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7347.654320] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7347.709226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7349.727243] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7352.431461] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7358.873384] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7362.738325] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7362.956120] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7366.737861] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7366.981763] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7370.218194] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7370.220249] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7370.223172] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7370.251539] Lustre: Skipped 3 previous similar messages [ 7370.257063] Lustre: Skipped 5 previous similar messages [ 7375.239868] LustreError: 211983:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7375.277132] LustreError: 211983:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 7375.339539] LustreError: 210772:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7375.358093] LustreError: 210772:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7375.451299] Lustre: server umount lustre-OST0000 complete [ 7386.529788] Lustre: server umount lustre-MDT0000 complete [ 7390.811628] LustreError: 209036:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763344059 with bad export cookie 9245164389958237314 [ 7390.821868] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7390.822712] LustreError: 209036:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7391.101320] Lustre: server umount lustre-MDT0001 complete [ 7401.387206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7406.486884] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7415.741174] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7420.620377] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7425.762989] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7429.099287] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7435.520348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7440.696696] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7445.699577] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7461.856714] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7461.871371] LustreError: Skipped 1 previous similar message [ 7461.879970] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7461.907456] Lustre: Skipped 2 previous similar messages [ 7461.923118] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7461.930342] Lustre: Skipped 4 previous similar messages [ 7466.959755] Lustre: server umount lustre-OST0000 complete [ 7485.407165] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7485.557508] Lustre: server umount lustre-MDT0000 complete [ 7488.381040] LustreError: 212847:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763344156 with bad export cookie 9245164389958238196 [ 7488.385806] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7488.389641] LustreError: 212847:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7488.606958] Lustre: server umount lustre-MDT0001 complete [ 7584.114426] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 7586.666670] Key type lgssc unregistered [ 7586.913070] LNet: 216200:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7586.920422] LNetError: 216200:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7586.932669] LNet: Removed LNI 192.168.201.116@tcp [ 7587.609191] Key type .llcrypt unregistered [ 7587.611075] Key type ._llcrypt unregistered [ 7615.182272] Key type ._llcrypt registered [ 7615.184895] Key type .llcrypt registered [ 7615.302956] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 7632.784880] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 7633.591198] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7633.849471] alg: No test for adler32 (adler32-zlib) [ 7634.898964] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 7635.089187] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 7636.759708] Key type lgssc registered [ 7638.092337] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7649.329797] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7657.836323] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7664.808580] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7672.542154] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7679.597734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7679.678892] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7679.693878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7680.958722] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7680.996746] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7681.133823] Lustre: lustre-MDT0000: new disk, initializing [ 7681.235109] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7681.264246] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7684.828665] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7694.735291] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7694.790446] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7694.922702] Lustre: 219529:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7694.989348] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7694.992631] Lustre: Skipped 1 previous similar message [ 7695.058297] Lustre: lustre-MDT0001: new disk, initializing [ 7695.142973] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7695.171686] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7695.175860] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7699.392522] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7707.147822] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7710.819841] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7717.421510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7717.475154] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7717.633780] Lustre: lustre-OST0000: new disk, initializing [ 7717.641690] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7717.716104] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7722.919359] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7727.153548] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7727.164066] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7727.285985] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7729.970228] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7733.977878] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7734.182339] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7737.681601] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7737.873743] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7742.950798] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7742.951658] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7742.953959] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7742.954259] LustreError: Skipped 1 previous similar message [ 7742.968656] Lustre: Skipped 1 previous similar message [ 7745.748136] LustreError: 222468:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7745.931633] Lustre: server umount lustre-OST0000 complete [ 7751.649349] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7751.661291] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7751.693578] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7751.696237] Lustre: Skipped 1 previous similar message [ 7753.184518] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7753.189171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7753.207537] Lustre: Skipped 1 previous similar message [ 7756.044932] LustreError: 222670:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7756.055941] LustreError: 222670:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7756.188231] Lustre: server umount lustre-MDT0000 complete [ 7760.331710] LustreError: 222671:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763344428 with bad export cookie 9616008814567697637 [ 7760.335655] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7760.347795] LustreError: 222671:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7760.563872] LustreError: 222872:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7760.570471] LustreError: 222872:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7760.768164] Lustre: server umount lustre-MDT0001 complete [ 7779.429941] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 7790.636509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7791.203616] LustreError: 224048:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7791.224670] LustreError: 224048:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7791.294339] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7796.155439] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7796.721196] LustreError: 224048:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7800.806147] LustreError: 224048:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7805.921773] LustreError: 224048:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7806.273990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7806.614759] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7810.916809] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7813.834684] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7821.220197] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7821.706468] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7827.503690] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7835.812175] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7835.897320] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7836.054498] Lustre: lustre-OST0001: new disk, initializing [ 7836.060777] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7841.877880] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7843.932034] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7843.938370] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7844.022396] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7852.313261] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7861.979564] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7881.227237] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 20:55:47 (1763344547) [ 7883.234019] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7883.240602] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7883.257890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7889.385851] LustreError: 227449:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 7889.392848] LustreError: 227449:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7889.481554] Lustre: server umount lustre-MDT0000 complete [ 7890.405182] LustreError: 225301:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7890.418955] LustreError: 225301:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7893.280258] LustreError: 224028:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763344561 with bad export cookie 9616008814567698519 [ 7893.288164] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7893.295400] LustreError: 224028:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7893.463847] LustreError: 227651:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 7893.468100] LustreError: 227651:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7893.660661] Lustre: server umount lustre-MDT0001 complete [ 7907.938854] LustreError: 227852:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7907.954521] LustreError: 227852:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7908.023704] Lustre: server umount lustre-OST0000 complete [ 7910.883155] Lustre: 217739:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763344563/real 1763344563] req@ffff91cd54091880 x1849000515972736/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763344579 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7910.898250] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7910.911673] Lustre: Skipped 3 previous similar messages [ 7911.802801] Lustre: server umount lustre-OST0001 complete [ 7917.537036] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 7929.413844] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 7939.671233] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7944.778366] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7949.585636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7954.534791] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7959.546752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7959.602879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7959.775872] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7959.799864] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7959.852065] Lustre: lustre-MDT0000: new disk, initializing [ 7959.947858] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7959.952267] Lustre: Skipped 1 previous similar message [ 7959.963508] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7962.926944] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7971.678874] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7971.741336] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7971.808484] Lustre: 230240:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7971.863678] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7971.866727] Lustre: Skipped 1 previous similar message [ 7971.938790] Lustre: lustre-MDT0001: new disk, initializing [ 7972.060779] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7972.066040] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7975.337424] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7981.794244] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7985.174197] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7991.896944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7991.952957] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7992.114782] Lustre: lustre-OST0000: new disk, initializing [ 7992.121507] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7992.195612] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7992.204898] Lustre: Skipped 1 previous similar message [ 7993.797405] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7993.811042] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7993.944243] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7998.122835] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 8006.089571] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8010.942799] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8011.223946] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8017.221430] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8017.586180] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8024.543956] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8024.544626] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8024.548684] LustreError: Skipped 1 previous similar message [ 8024.555724] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8024.555732] Lustre: Skipped 4 previous similar messages [ 8024.610481] Lustre: Skipped 1 previous similar message [ 8034.271114] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8034.333142] LustreError: 233179:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8034.341316] LustreError: 233179:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 8034.564427] Lustre: server umount lustre-OST0000 complete [ 8039.906770] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8039.918611] Lustre: Skipped 4 previous similar messages [ 8053.215474] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8053.437758] Lustre: server umount lustre-MDT0000 complete [ 8055.265786] LustreError: 230246:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8055.281744] LustreError: 230246:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8056.990763] LustreError: 230232:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763344725 with bad export cookie 9616008814567700661 [ 8056.994449] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8057.009056] LustreError: 230232:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8057.356199] Lustre: server umount lustre-MDT0001 complete [ 8065.448355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8065.885527] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8069.396827] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 8076.263440] LustreError: 234061:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8076.277914] LustreError: 234061:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8078.373826] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8082.152470] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 8087.003574] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8090.494704] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8096.334182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8101.782573] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 8106.980350] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8252.385610] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8252.396830] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8252.405132] Lustre: Skipped 2 previous similar messages [ 8252.416263] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8252.420140] Lustre: Skipped 4 previous similar messages [ 8258.245761] LustreError: 236359:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8258.248395] LustreError: 236359:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 8258.376233] Lustre: server umount lustre-OST0000 complete [ 8263.137354] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8267.864432] Lustre: server umount lustre-MDT0000 complete [ 8269.794661] LustreError: 234056:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8269.811524] LustreError: 234056:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8271.255134] LustreError: 234041:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763344939 with bad export cookie 9616008814567701543 [ 8271.263254] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8271.263648] LustreError: 234041:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8271.522263] Lustre: server umount lustre-MDT0001 complete [ 8376.118460] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 8379.079587] Key type lgssc unregistered [ 8379.434251] LNet: 237331:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8379.447095] LNetError: 237331:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8380.456788] LNet: Removed LNI 192.168.201.116@tcp [ 8381.447887] Key type .llcrypt unregistered [ 8381.449539] Key type ._llcrypt unregistered [ 8403.343326] Key type ._llcrypt registered [ 8403.346017] Key type .llcrypt registered [ 8403.427249] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 8415.920142] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 8416.777198] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8416.828264] alg: No test for adler32 (adler32-zlib) [ 8417.805634] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 8417.978218] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 8419.663194] Key type lgssc registered [ 8420.427789] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8426.561814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8431.663031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8436.265229] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8440.636926] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8453.155478] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 8460.205873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8460.244472] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8460.252949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8461.414241] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8461.459226] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8461.536444] Lustre: lustre-MDT0000: new disk, initializing [ 8461.601581] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8461.625554] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8464.182447] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 8472.386983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8472.441914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8472.511831] Lustre: 241350:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8472.576853] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8472.579592] Lustre: Skipped 1 previous similar message [ 8472.624891] Lustre: lustre-MDT0001: new disk, initializing [ 8472.703779] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8472.741049] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8472.751312] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8475.306522] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 8479.440033] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8484.727849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8484.775647] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8484.972525] Lustre: lustre-OST0000: new disk, initializing [ 8484.984393] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8485.065907] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8488.111604] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8488.119711] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8488.209050] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8489.366358] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 8498.057632] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8498.111116] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8498.244694] Lustre: lustre-OST0001: new disk, initializing [ 8498.248792] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8498.335888] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8502.187575] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 8504.467785] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8504.473821] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8504.560863] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8512.096928] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8519.182373] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8535.525771] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8535.531912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8535.543193] Lustre: Skipped 3 previous similar messages [ 8535.546914] Lustre: Skipped 3 previous similar messages [ 8540.647016] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8540.665185] Lustre: Skipped 3 previous similar messages [ 8540.868763] LustreError: 244871:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 8542.951226] LustreError: 244871:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 8542.968485] LustreError: 244871:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8543.064364] Lustre: server umount lustre-MDT0000 complete [ 8546.515077] LustreError: 241341:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763345214 with bad export cookie 16109020473043014858 [ 8546.521543] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8546.676316] LustreError: 245074:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 8546.678671] LustreError: 245074:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 8546.787478] Lustre: server umount lustre-MDT0001 complete [ 8560.161655] LustreError: 245274:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8560.187771] LustreError: 245274:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 8560.249105] Lustre: server umount lustre-OST0000 complete [ 8574.179505] LustreError: 245476:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 8574.194081] LustreError: 245476:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8574.328682] Lustre: server umount lustre-OST0001 complete [ 8584.889221] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8592.505678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8593.565612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8601.148227] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 8607.719458] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8608.543045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 8623.038532] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 8630.133362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8630.160568] Lustre: lustre-MDT0000: reset Object Index mappings [ 8630.545824] LustreError: 248789:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8630.558445] LustreError: 248789:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 8630.613227] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8633.797393] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 8635.876974] LustreError: 248789:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8640.995159] LustreError: 248789:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8641.252759] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8641.264598] Lustre: lustre-MDT0001: reset Object Index mappings [ 8641.595462] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8645.547476] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 8649.111114] Lustre: 249898:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8655.122307] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8656.426764] LustreError: 250253:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8656.452272] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8660.511542] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 8661.486138] LustreError: 250333:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8668.409763] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8668.657334] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8668.661347] Lustre: Skipped 1 previous similar message [ 8670.421371] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 8673.559764] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 8679.149065] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8681.880853] Lustre: 251749:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8692.195256] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8692.207927] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8692.209465] Lustre: Skipped 3 previous similar messages [ 8692.220973] Lustre: Skipped 3 previous similar messages [ 8696.548449] LustreError: 251852:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 8696.552341] LustreError: 251852:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8696.613562] Lustre: server umount lustre-MDT0000 complete [ 8697.321390] LustreError: 248788:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8697.330086] LustreError: 248788:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 8699.151336] LustreError: 251853:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763345367 with bad export cookie 16109020473043021648 [ 8699.164019] LustreError: 251853:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8699.164366] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8699.401933] Lustre: server umount lustre-MDT0001 complete [ 8711.757439] Lustre: server umount lustre-OST0000 complete [ 8725.494492] LustreError: 252458:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 8725.514572] LustreError: 252458:0:(obd_class.h:479:obd_check_dev()) Skipped 19 previous similar messages [ 8725.839991] Lustre: server umount lustre-OST0001 complete [ 8744.802828] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 21:10:10 (1763345410) [ 8757.266551] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 8764.128774] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 8771.578181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8777.846877] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8783.171572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8788.747335] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8794.558824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8794.611343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8794.808846] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8794.834128] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8794.889667] Lustre: lustre-MDT0000: new disk, initializing [ 8794.954048] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8794.977875] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8798.856578] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 8807.895305] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8807.960980] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8808.028568] Lustre: 255983:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8808.089841] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8808.092910] Lustre: Skipped 1 previous similar message [ 8808.161390] Lustre: lustre-MDT0001: new disk, initializing [ 8808.299922] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8808.310840] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8811.477718] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 8818.613725] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8822.175562] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8828.796616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8828.847145] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8829.117781] Lustre: lustre-OST0000: new disk, initializing [ 8829.123468] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8830.740517] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8830.773714] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8830.997384] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8835.778610] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 8842.812803] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8846.778538] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8847.026234] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8850.560416] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8850.742643] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8856.550421] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8856.553820] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8856.562891] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8858.314175] LustreError: 258924:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8858.319409] LustreError: 258924:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8858.420085] Lustre: server umount lustre-OST0000 complete [ 8861.664956] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8861.669352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8861.672156] Lustre: Skipped 2 previous similar messages [ 8861.675897] Lustre: Skipped 1 previous similar message [ 8867.897851] Lustre: server umount lustre-MDT0000 complete [ 8870.751941] LustreError: 258925:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763345539 with bad export cookie 16109020473043023741 [ 8870.760632] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8870.761780] LustreError: 258925:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8870.999305] Lustre: server umount lustre-MDT0001 complete [ 8884.393357] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 8892.401796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8892.685208] LustreError: 260503:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8892.749375] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8892.752783] Lustre: Skipped 2 previous similar messages [ 8895.827345] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 8902.943585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8903.138348] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 8903.146967] Lustre: Skipped 2 previous similar messages [ 8906.770820] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 8909.376049] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8915.550666] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8921.604942] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 8930.473613] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8930.613117] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8930.888624] Lustre: lustre-OST0001: new disk, initializing [ 8930.893148] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8932.401733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8932.418797] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8932.653135] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8936.571531] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 8946.744263] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8950.989401] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8960.257269] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 8960.269576] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 8967.138161] LustreError: 264009:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 8967.143503] LustreError: 264009:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 8967.197138] Lustre: server umount lustre-MDT0000 complete [ 8968.674173] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8968.676711] LustreError: 260499:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8968.680678] Lustre: Skipped 2 previous similar messages [ 8968.694236] LustreError: 260499:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 8969.729761] LustreError: 260483:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763345638 with bad export cookie 16109020473043024623 [ 8969.732884] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8969.739830] LustreError: 260483:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8969.976141] Lustre: server umount lustre-MDT0001 complete [ 8983.125697] Lustre: server umount lustre-OST0000 complete [ 8996.524794] Lustre: server umount lustre-OST0001 complete [ 9012.723896] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 9024.066779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9024.442773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9024.448142] Lustre: Skipped 3 previous similar messages [ 9028.229211] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 9033.709982] LustreError: 265788:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9033.726746] LustreError: 265788:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9036.033396] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9040.278337] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 9043.019229] Lustre: 266900:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9049.677917] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9055.608520] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 9063.158652] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9068.364441] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 9069.182984] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 9069.238234] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 9075.491507] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9078.773268] Lustre: 268757:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9088.339402] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 9088.356115] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 9097.058550] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 21:16:03 (1763345763) [ 9113.569120] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9113.572960] LustreError: Skipped 1 previous similar message [ 9113.575107] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9113.580561] Lustre: Skipped 1 previous similar message [ 9113.584048] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9116.997846] LustreError: 269466:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 9117.000353] LustreError: 269466:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 9117.098300] Lustre: server umount lustre-MDT0000 complete [ 9120.501593] LustreError: 265769:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763345788 with bad export cookie 16109020473043030916 [ 9120.511814] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9120.522119] LustreError: 265769:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9126.934089] Lustre: server umount lustre-MDT0001 complete [ 9145.311143] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9145.441904] Lustre: server umount lustre-OST0000 complete [ 9148.892818] Lustre: server umount lustre-OST0001 complete [ 9154.743878] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 9165.205894] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 9175.027493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9180.849767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9185.714994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9190.856368] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9205.015624] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 9212.496596] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9212.547808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9212.767682] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9212.808186] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9212.876620] Lustre: lustre-MDT0000: new disk, initializing [ 9212.974410] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9215.892669] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 9224.313346] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9224.365695] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9224.431255] Lustre: 272956:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9224.679289] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9227.990468] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 9237.614261] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9237.694169] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9239.656631] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9243.255878] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 9252.562715] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9252.618735] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9254.631247] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9258.349584] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 9267.754836] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9271.307214] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9271.309828] Lustre: Skipped 1 previous similar message [ 9289.770567] Lustre: DEBUG MARKER: == conf-sanity test 160: MGC updates failnodes from all participants ========================================================== 21:19:15 (1763345955) [ 9295.844584] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9295.845664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9295.864506] Lustre: Skipped 9 previous similar messages [ 9295.872494] Lustre: Skipped 7 previous similar messages [ 9297.864069] Lustre: server umount lustre-MDT0000 complete [ 9300.964255] LustreError: 272964:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9300.982880] LustreError: 272964:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 9301.252114] LustreError: 272949:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763345969 with bad export cookie 16109020473043038777 [ 9301.252974] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9301.256513] LustreError: 272949:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9301.546631] Lustre: server umount lustre-MDT0001 complete [ 9315.368221] Lustre: server umount lustre-OST0000 complete [ 9328.935363] Lustre: server umount lustre-OST0001 complete [ 9346.110663] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 9355.552864] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 9364.867014] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9372.927159] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9381.487308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9389.111039] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9398.076186] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9398.176094] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9398.484801] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9398.534374] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9398.538321] Lustre: Skipped 4 previous similar messages [ 9398.616207] Lustre: lustre-MDT0000: new disk, initializing [ 9398.618975] Lustre: Skipped 3 previous similar messages [ 9398.789857] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9398.793125] Lustre: Skipped 7 previous similar messages [ 9398.832368] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9398.848537] Lustre: Skipped 3 previous similar messages [ 9403.306898] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 9413.620900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9413.713304] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9413.781798] Lustre: 280022:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9414.071204] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9414.077498] Lustre: Skipped 2 previous similar messages [ 9417.718724] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 9424.719666] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9428.239447] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9435.165576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9435.266868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9437.278223] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9441.695995] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 9450.282604] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9453.003335] Lustre: Mounted lustre-client [ 9454.243120] LustreError: 282524:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff91ce5095f000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 9454.288384] LustreError: 282524:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 9454.291653] LustreError: 282524:0:(obd_class.h:479:obd_check_dev()) Skipped 51 previous similar messages [ 9454.365697] Lustre: Unmounted lustre-client [ 9467.873088] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9467.881062] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9467.911437] Lustre: Skipped 1 previous similar message [ 9467.921032] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9467.936527] Lustre: Skipped 3 previous similar messages [ 9481.183486] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9481.391646] Lustre: server umount lustre-OST0000 complete [ 9485.795319] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9485.808339] LustreError: Skipped 1 previous similar message [ 9491.906343] Lustre: server umount lustre-MDT0000 complete [ 9496.499675] LustreError: 280014:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763346164 with bad export cookie 16109020473043040303 [ 9496.504739] LustreError: 280014:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9496.506789] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9496.989207] Lustre: server umount lustre-MDT0001 complete [ 9507.743902] LNet: 1 local NIs in recovery (showing 1): 192.168.201.116@tcp [ 9509.104578] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 9512.098040] Key type lgssc unregistered [ 9512.445718] LNet: 283808:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9512.456771] LNetError: 283808:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9582.128928] LNet: Removed LNI 192.168.201.116@tcp [ 9583.151460] Key type .llcrypt unregistered [ 9583.162344] Key type ._llcrypt unregistered [ 9600.176097] Key type ._llcrypt registered [ 9600.177980] Key type .llcrypt registered [ 9600.265350] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 9616.821592] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 9617.657931] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9617.686139] alg: No test for adler32 (adler32-zlib) [ 9618.852281] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 9619.102511] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 9620.759246] Key type lgssc registered [ 9621.991465] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9631.181660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9639.137764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9648.394084] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9656.805416] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9658.982655] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 21:25:24 (1763346324) [ 9666.631937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9666.726471] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9666.755945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9668.139158] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9668.183742] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9668.280664] Lustre: lustre-MDT0000: new disk, initializing [ 9668.363280] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9668.379151] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9673.569349] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 9685.718717] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9685.791693] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9685.881129] Lustre: 287002:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9685.960426] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9685.969662] Lustre: Skipped 1 previous similar message [ 9686.134937] Lustre: lustre-MDT0001: new disk, initializing [ 9686.244624] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9686.292491] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9686.315100] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9690.139459] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 9697.810565] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9701.454661] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9707.874094] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9707.946364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9708.277339] Lustre: lustre-OST0000: new disk, initializing [ 9708.284751] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9708.401808] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9713.651092] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 9716.799316] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9716.808891] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9716.927482] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9721.427732] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9732.582337] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9732.585771] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9732.599526] LustreError: Skipped 1 previous similar message [ 9732.606497] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9732.617766] Lustre: Skipped 1 previous similar message [ 9737.675722] LustreError: 289447:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 9737.843961] Lustre: server umount lustre-OST0000 complete [ 9742.817548] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9742.821730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9742.840468] Lustre: Skipped 2 previous similar messages [ 9747.627587] LustreError: 289649:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 9747.630920] LustreError: 289649:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9747.721316] Lustre: server umount lustre-MDT0000 complete [ 9747.938734] LustreError: 287013:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9747.948151] LustreError: 287013:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 9750.904226] LustreError: 286994:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763346419 with bad export cookie 1118907102309654694 [ 9750.912197] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9750.920090] LustreError: 286994:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9751.073751] LustreError: 289850:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9751.081847] LustreError: 289850:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9751.229325] Lustre: server umount lustre-MDT0001 complete [ 9761.360823] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 9763.449391] Key type lgssc unregistered [ 9763.694580] LNet: 290382:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9763.708904] LNetError: 290382:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9763.729223] LNet: Removed LNI 192.168.201.116@tcp [ 9764.390759] Key type .llcrypt unregistered [ 9764.392597] Key type ._llcrypt unregistered [ 9849.498234] Key type ._llcrypt registered [ 9849.499682] Key type .llcrypt registered [ 9849.613768] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 9849.880529] Key type .llcrypt unregistered [ 9849.882214] Key type ._llcrypt unregistered [ 9920.832482] Key type ._llcrypt registered [ 9920.836503] Key type .llcrypt registered [ 9920.942722] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 9921.226794] Key type .llcrypt unregistered [ 9921.230319] Key type ._llcrypt unregistered [ 9951.084967] Key type ._llcrypt registered [ 9951.086671] Key type .llcrypt registered [ 9951.219071] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 9951.471554] Key type .llcrypt unregistered [ 9951.477284] Key type ._llcrypt unregistered [ 9974.826861] Key type ._llcrypt registered [ 9974.828646] Key type .llcrypt registered [ 9974.902825] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 9975.109322] Key type .llcrypt unregistered [ 9975.114733] Key type ._llcrypt unregistered [10033.847155] Key type ._llcrypt registered [10033.848839] Key type .llcrypt registered [10033.943051] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [10034.166643] Key type .llcrypt unregistered [10034.168458] Key type ._llcrypt unregistered [10056.041839] Key type ._llcrypt registered [10056.043532] Key type .llcrypt registered [10056.119688] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [10056.314305] Key type .llcrypt unregistered [10056.316292] Key type ._llcrypt unregistered [10080.114996] Key type ._llcrypt registered [10080.116445] Key type .llcrypt registered [10080.214667] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [10080.475909] Key type .llcrypt unregistered [10080.480295] Key type ._llcrypt unregistered [10106.024591] Key type ._llcrypt registered [10106.026470] Key type .llcrypt registered [10106.136293] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [10106.412410] Key type .llcrypt unregistered [10106.413921] Key type ._llcrypt unregistered [10129.730937] Key type ._llcrypt registered [10129.732885] Key type .llcrypt registered [10129.833826] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [10130.039331] Key type .llcrypt unregistered [10130.041053] Key type ._llcrypt unregistered [10156.718791] Key type ._llcrypt registered [10156.724227] Key type .llcrypt registered [10156.881757] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [10157.147439] Key type .llcrypt unregistered [10157.149073] Key type ._llcrypt unregistered [10209.962739] Key type ._llcrypt registered [10209.964436] Key type .llcrypt registered [10210.100519] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [10210.463501] Key type .llcrypt unregistered [10210.469548] Key type ._llcrypt unregistered [10234.552579] Key type ._llcrypt registered [10234.555081] Key type .llcrypt registered [10234.622169] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [10234.821524] Key type .llcrypt unregistered [10234.823242] Key type ._llcrypt unregistered [10261.262310] Key type ._llcrypt registered [10261.268181] Key type .llcrypt registered [10261.398526] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [10261.666248] Key type .llcrypt unregistered [10261.671159] Key type ._llcrypt unregistered [10309.033928] Key type ._llcrypt registered [10309.035939] Key type .llcrypt registered [10309.128483] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [10322.026320] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [10322.715962] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10322.773654] alg: No test for adler32 (adler32-zlib) [10323.817477] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [10324.003520] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [10325.663616] Key type lgssc registered [10326.636751] Lustre: Echo OBD driver; http://www.lustre.org/ [10333.675376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10339.046722] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10344.074043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10349.848721] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10351.499921] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 10147 sec ========= 21:36:57 (1763347017) [10353.404486] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 21:36:59 (1763347019) === [10356.738711] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 21:37:02 (1763347022) === [10394.877562] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [10397.309316] Key type lgssc unregistered [10397.562414] LNet: 312284:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10397.568109] LNetError: 312284:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10397.589249] LNet: Removed LNI 192.168.201.116@tcp [10398.372956] Key type .llcrypt unregistered [10398.374746] Key type ._llcrypt unregistered