[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 468243567 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003373] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005012] kvm-guest: setup PV IPIs [ 0.007803] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009009] pid_max: default: 32768 minimum: 301 [ 0.010134] LSM: Security Framework initializing [ 0.011049] Yama: becoming mindful. [ 0.012030] SELinux: Initializing. [ 0.013052] *** VALIDATE selinux *** [ 0.020803] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026186] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027113] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028091] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030074] *** VALIDATE tmpfs *** [ 0.031357] *** VALIDATE proc *** [ 0.032152] *** VALIDATE cgroup *** [ 0.033009] *** VALIDATE cgroup2 *** [ 0.034264] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035140] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037032] Spectre V2 : User space: Vulnerable [ 0.038009] Speculative Store Bypass: Vulnerable [ 0.041348] debug: unmapping init [mem 0xffffffff8b659000-0xffffffff8b660fff] [ 0.043171] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044695] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045026] ... version: 2 [ 0.046009] ... bit width: 48 [ 0.047010] ... generic registers: 4 [ 0.048013] ... value mask: 0000ffffffffffff [ 0.049018] ... max period: 00007fffffffffff [ 0.050013] ... fixed-purpose events: 3 [ 0.051012] ... event mask: 000000070000000f [ 0.053208] rcu: Hierarchical SRCU implementation. [ 0.055332] smp: Bringing up secondary CPUs ... [ 0.056602] x86: Booting SMP configuration: [ 0.057028] .... node #0, CPUs: #1 #2 #3 [ 0.060593] smp: Brought up 1 node, 4 CPUs [ 0.062015] smpboot: Max logical packages: 1 [ 0.063020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.222602] node 0 deferred pages initialised in 159ms [ 0.226155] devtmpfs: initialized [ 0.227220] x86/mm: Memory block size: 128MB [ 0.229309] gcov: version magic: 0x41383552 [ 0.231371] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.232078] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.233000] pinctrl core: initialized pinctrl subsystem [ 0.234218] [ 0.234770] ************************************************************* [ 0.237021] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.239015] ** ** [ 0.241015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.243014] ** ** [ 0.245013] ** This means that this kernel is built to expose internal ** [ 0.247016] ** IOMMU data structures, which may compromise security on ** [ 0.250016] ** your system. ** [ 0.251017] ** ** [ 0.253010] ** If you see this message and you are not debugging the ** [ 0.254008] ** kernel, report this immediately to your vendor! ** [ 0.256010] ** ** [ 0.257007] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.259013] ************************************************************* [ 0.260658] NET: Registered protocol family 16 [ 0.262344] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.263043] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.266070] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.269146] cpuidle: using governor menu [ 0.271360] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.273495] PCI: Using configuration type 1 for base access [ 0.276158] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.283126] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.285046] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.288074] cryptd: max_cpu_qlen set to 1000 [ 0.292888] ACPI: Added _OSI(Module Device) [ 0.294024] ACPI: Added _OSI(Processor Device) [ 0.295015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.296011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.299000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.305588] ACPI: Interpreter enabled [ 0.307082] ACPI: PM: (supports S0 S3 S4 S5) [ 0.309013] ACPI: Using IOAPIC for interrupt routing [ 0.310112] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.313471] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.322328] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.324056] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.327025] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.330102] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.335044] acpiphp: Slot [2] registered [ 0.336107] acpiphp: Slot [5] registered [ 0.337106] acpiphp: Slot [6] registered [ 0.338084] acpiphp: Slot [7] registered [ 0.338915] acpiphp: Slot [8] registered [ 0.341110] acpiphp: Slot [9] registered [ 0.342144] acpiphp: Slot [10] registered [ 0.342977] acpiphp: Slot [3] registered [ 0.344089] acpiphp: Slot [4] registered [ 0.344911] acpiphp: Slot [11] registered [ 0.345062] acpiphp: Slot [12] registered [ 0.346091] acpiphp: Slot [13] registered [ 0.346966] acpiphp: Slot [14] registered [ 0.348053] acpiphp: Slot [15] registered [ 0.349115] acpiphp: Slot [16] registered [ 0.350110] acpiphp: Slot [17] registered [ 0.350973] acpiphp: Slot [18] registered [ 0.352055] acpiphp: Slot [19] registered [ 0.353077] acpiphp: Slot [20] registered [ 0.355110] acpiphp: Slot [21] registered [ 0.356107] acpiphp: Slot [22] registered [ 0.356982] acpiphp: Slot [23] registered [ 0.358055] acpiphp: Slot [24] registered [ 0.358778] acpiphp: Slot [25] registered [ 0.359052] acpiphp: Slot [26] registered [ 0.359849] acpiphp: Slot [27] registered [ 0.361084] acpiphp: Slot [28] registered [ 0.362139] acpiphp: Slot [29] registered [ 0.364119] acpiphp: Slot [30] registered [ 0.365105] acpiphp: Slot [31] registered [ 0.366097] PCI host bridge to bus 0000:00 [ 0.368029] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.370031] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.372022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.374026] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.376029] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.378027] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.379190] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.382325] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.384975] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.393675] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.398063] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.401027] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.403022] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.406025] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.408588] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.411816] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.413038] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.416617] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.421019] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.431022] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.436014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.441624] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.447020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.453022] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.474019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.484701] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.491017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.499023] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.519022] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.529359] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.535018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.543019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.554022] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.563258] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.569025] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.576017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.590023] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.599423] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.611018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.621020] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.637024] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.646312] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.656026] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.663020] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.683025] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.692539] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.695403] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.697337] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.699952] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.702241] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.706660] iommu: Default domain type: Passthrough [ 0.708451] SCSI subsystem initialized [ 0.710198] ACPI: bus type USB registered [ 0.711086] usbcore: registered new interface driver usbfs [ 0.712108] usbcore: registered new interface driver hub [ 0.714167] usbcore: registered new device driver usb [ 0.716201] pps_core: LinuxPPS API ver. 1 registered [ 0.718014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.721088] PTP clock support registered [ 0.723211] EDAC MC: Ver: 3.0.0 [ 0.724468] PCI: Using ACPI for IRQ routing [ 0.726889] NetLabel: Initializing [ 0.728016] NetLabel: domain hash size = 128 [ 0.729009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.731093] NetLabel: unlabeled traffic allowed by default [ 0.733176] vgaarb: loaded [ 0.735320] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.737018] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.746000] clocksource: Switched to clocksource kvm-clock [ 0.842301] VFS: Disk quotas dquot_6.6.0 [ 0.843883] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.846538] *** VALIDATE ramfs *** [ 0.847517] *** VALIDATE hugetlbfs *** [ 0.848941] pnp: PnP ACPI init [ 0.851340] pnp: PnP ACPI: found 6 devices [ 0.869202] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.872743] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.874464] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.876159] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.878094] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.880153] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.883122] NET: Registered protocol family 2 [ 0.885844] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.890814] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.894439] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.899679] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.903177] TCP: Hash tables configured (established 65536 bind 65536) [ 0.905564] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.908246] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.911025] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.913602] NET: Registered protocol family 1 [ 0.915731] RPC: Registered named UNIX socket transport module. [ 0.918247] RPC: Registered udp transport module. [ 0.920207] RPC: Registered tcp transport module. [ 0.922197] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.924844] NET: Registered protocol family 44 [ 0.926722] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.929182] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.931533] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.934221] PCI: CLS 0 bytes, default 64 [ 0.935762] Unpacking initramfs... [ 2.334702] debug: unmapping init [mem 0xffff9ae7fcc54000-0xffff9ae7fffbffff] [ 2.340511] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.342042] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.346460] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.815446] Initialise system trusted keyrings [ 2.817244] Key type blacklist registered [ 2.819071] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.826347] zbud: loaded [ 2.829178] *** VALIDATE nfs *** [ 2.830075] *** VALIDATE nfs4 *** [ 2.831199] pstore: using deflate compression [ 2.834550] Platform Keyring initialized [ 2.920312] NET: Registered protocol family 38 [ 2.921601] Key type asymmetric registered [ 2.923195] Asymmetric key parser 'x509' registered [ 2.924898] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.928153] io scheduler mq-deadline registered [ 2.929565] io scheduler kyber registered [ 2.930554] io scheduler bfq registered [ 2.931839] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.933604] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.935883] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.938621] ACPI: Power Button [PWRF] [ 2.943842] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.950204] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.964390] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.972142] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.984569] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.012321] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.039845] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.043858] Non-volatile memory driver v1.3 [ 3.045128] Linux agpgart interface v0.103 [ 3.073191] virtio_blk virtio1: [vda] 150616 512-byte logical blocks (77.1 MB/73.5 MiB) [ 3.077559] vda: detected capacity change from 0 to 77115392 [ 3.094542] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.096364] vdb: detected capacity change from 0 to 1073741824 [ 3.106662] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.108939] vdc: detected capacity change from 0 to 2621440000 [ 3.120987] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.124339] vdd: detected capacity change from 0 to 2621440000 [ 3.137852] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.141059] vde: detected capacity change from 0 to 4294967296 [ 3.159157] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.162290] vdf: detected capacity change from 0 to 4294967296 [ 3.169555] libphy: Fixed MDIO Bus: probed [ 3.177706] usbcore: registered new interface driver usbserial_generic [ 3.180042] usbserial: USB Serial support registered for generic [ 3.182237] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.186372] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.188259] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.191045] mousedev: PS/2 mouse device common for all mice [ 3.194745] rtc_cmos 00:05: RTC can wake from S4 [ 3.198097] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.205148] rtc_cmos 00:05: registered as rtc0 [ 3.205747] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.207030] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.213798] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.214437] intel_pstate: CPU model not supported [ 3.221415] hid: raw HID events driver (C) Jiri Kosina [ 3.223950] usbcore: registered new interface driver usbhid [ 3.226158] usbhid: USB HID core driver [ 3.227347] drop_monitor: Initializing network drop monitor service [ 3.230110] Initializing XFRM netlink socket [ 3.232157] NET: Registered protocol family 10 [ 3.234943] Segment Routing with IPv6 [ 3.236976] NET: Registered protocol family 17 [ 3.239443] mpls_gso: MPLS GSO support [ 3.245397] RAS: Correctable Errors collector initialized. [ 3.247414] AVX version of gcm_enc/dec engaged. [ 3.248788] AES CTR mode by8 optimization enabled [ 3.315988] sched_clock: Marking stable (3315962155, 0)->(4119940755, -803978600) [ 3.318875] registered taskstats version 1 [ 3.320424] Loading compiled-in X.509 certificates [ 3.322204] zswap: loaded using pool lzo/zbud [ 3.343302] Key type big_key registered [ 3.355556] Key type encrypted registered [ 3.357470] ima: No TPM chip found, activating TPM-bypass! [ 3.359664] ima: Allocated hash algorithm: sha1 [ 3.361539] ima: No architecture policies found [ 3.363415] evm: Initialising EVM extended attributes: [ 3.365545] evm: security.selinux [ 3.366923] evm: security.ima [ 3.368139] evm: security.capability [ 3.369473] evm: HMAC attrs: 0x1 [ 3.372348] rtc_cmos 00:05: setting system clock to 2026-10-02 16:34:42 UTC (1790958882) [ 3.379415] debug: unmapping init [mem 0xffffffff8c603000-0xffffffff8c7fffff] [ 3.382165] debug: unmapping init [mem 0xffffffff8b382000-0xffffffff8b658fff] [ 3.390071] Write protecting the kernel read-only data: 28672k [ 3.393391] debug: unmapping init [mem 0xffffffff89a03000-0xffffffff89bfffff] [ 3.396453] debug: unmapping init [mem 0xffffffff8a314000-0xffffffff8a3fffff] [ 3.427381] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.434858] systemd[1]: Detected virtualization kvm. [ 3.436548] systemd[1]: Detected architecture x86-64. [ 3.437752] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.463393] systemd[1]: No hostname configured. [ 3.464724] systemd[1]: Set hostname to . [ 3.466916] random: systemd: uninitialized urandom read (16 bytes read) [ 3.469739] systemd[1]: Initializing machine ID from random generator. [ 3.520554] random: ln: uninitialized urandom read (6 bytes read) [ 3.614213] random: systemd: uninitialized urandom read (16 bytes read) [ 3.616737] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.621136] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.625286] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. Starting Apply Kernel Variables... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Paths. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... [ OK ] Reached target Swap. [ OK ] Reached target Timers. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.234940] device-mapper: uevent: version 1.0.3 [ 4.237090] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] [ 4.555460] random: fast init done Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.861610] virtio_net virtio0 ens2: renamed from eth0 [ 4.943853] scsi host0: ata_piix [ 4.957996] scsi host1: ata_piix [ 4.959537] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.962111] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.636375] dracut-initqueue[583]: RTNETLINK answers: File exists [ 9.877453] random: crng init done [ 9.878724] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.154424] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.149684] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.355192] SELinux: Disabled at runtime. [ 11.410726] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.421145] systemd[1]: Detected virtualization kvm. [ 11.422861] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.821933] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.824568] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.828937] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.831703] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.834380] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.841618] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.845574] systemd[1]: Reached target rpc_pipefs.target. [ OK ] Reached target rpc_pipefs.target. Activating swap /dev/disk/by-label/SWAP... Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Kernel Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Kernel Debug File System... [ 11.874062] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting POSIX Message Queue File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Control Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice User and Session Slice. Mounting Huge Pages File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Stopped target Initrd Root File System. Starting Apply Kernel Variables... [ OK ] Created slice system-getty.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. Starting udev Coldplug all Devices... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.221164] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.453544] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.556576] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.620185] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.634748] EDAC sbridge: Ver: 1.1.2 [ 14.008626] Key type dns_resolver registered [ 14.294157] NFS: Registering the id_resolver key type [ 14.295979] Key type id_resolver registered [ 14.297333] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Login Service... Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg221-server login: [ 50.134014] hrtimer: interrupt took 2159282 ns [ 51.604565] libcfs: loading out-of-tree module taints kernel. [ 51.640412] Key type ._llcrypt registered [ 51.642131] Key type .llcrypt registered [ 51.700594] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [ 64.595217] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 65.500307] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 65.514672] alg: No test for adler32 (adler32-zlib) [ 66.719641] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 67.212516] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 68.903226] Key type lgssc registered [ 69.966869] Lustre: Echo OBD driver; http://www.lustre.org/ [ 81.412768] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 104.783436] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 111.335553] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 111.349853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 112.463420] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 112.485919] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 112.539693] Lustre: lustre-MDT0000: new disk, initializing [ 112.577733] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 112.588543] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 114.980149] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 123.556403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 123.614759] Lustre: 6505:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 123.645210] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 123.648609] Lustre: Skipped 1 previous similar message [ 123.716736] Lustre: lustre-MDT0001: new disk, initializing [ 123.766100] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 123.789335] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 123.799084] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 125.885298] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 128.845409] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 134.827597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 135.082665] Lustre: lustre-OST0000: new disk, initializing [ 135.086082] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 135.092381] Lustre: 8436:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 135.143234] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 135.678270] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 135.683779] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 135.750688] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 138.678107] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 147.960131] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 148.036653] Lustre: lustre-OST0001: new disk, initializing [ 148.039455] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 148.042217] Lustre: 9510:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 148.080993] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 151.232260] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 156.689140] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 156.696047] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 156.749038] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 160.451974] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 164.647973] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 169.055291] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing check_logdir /tmp/testlogs/ [ 172.639817] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing yml_node [ 174.871950] Lustre: DEBUG MARKER: Client: 2.17.59.2 [ 176.374439] Lustre: DEBUG MARKER: MDS: 2.17.59.2 [ 177.776458] Lustre: DEBUG MARKER: OSS: 2.17.59.2 [ 178.623127] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Oct 2 12:37:37 EDT 2026 [ 188.013278] Lustre: DEBUG MARKER: only running tests: 121-999 [ 188.897938] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 189.658248] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 202.720606] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 202.720727] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 202.729690] Lustre: Skipped 1 previous similar message [ 202.734355] Lustre: Skipped 3 previous similar messages [ 207.841524] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 207.849292] Lustre: Skipped 3 previous similar messages [ 208.833030] Lustre: server umount lustre-MDT0000 complete [ 210.883485] LustreError: 7915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790959090 with bad export cookie 1445685136879292771 [ 210.883741] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 210.896643] LustreError: 7915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 212.960311] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 212.962052] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 212.972659] Lustre: Skipped 3 previous similar messages [ 212.979723] Lustre: Skipped 1 previous similar message [ 215.968557] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 215.972285] Lustre: Skipped 1 previous similar message [ 221.088034] LustreError: 9518:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 221.088118] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 221.095144] LustreError: 9518:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 221.105501] Lustre: Skipped 1 previous similar message [ 225.247424] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 225.358089] Lustre: server umount lustre-MDT0001 complete [ 227.723978] Lustre: server umount lustre-OST0000 complete [ 229.733339] Lustre: server umount lustre-OST0001 complete [ 233.447912] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [ 238.173500] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 258.202062] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 258.377769] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 258.395372] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 258.450596] Lustre: lustre-MDT0000: new disk, initializing [ 258.533321] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 258.552576] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 260.833966] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 266.756715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 266.807744] Lustre: 15102:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 266.840642] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 266.845825] Lustre: Skipped 1 previous similar message [ 266.892059] Lustre: lustre-MDT0001: new disk, initializing [ 266.958781] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 266.993895] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 267.005636] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 269.677866] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 275.849649] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 279.088883] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 283.102753] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 283.289632] Lustre: lustre-OST0000: new disk, initializing [ 283.293064] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 283.296178] Lustre: 17069:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 283.362056] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 284.704309] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 284.708395] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 284.761952] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 286.772237] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 292.643045] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 295.380811] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 295.508412] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 298.210066] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 298.324932] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 300.002053] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 300.003124] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 300.007080] LustreError: Skipped 1 previous similar message [ 300.010092] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 300.013908] Lustre: Skipped 1 previous similar message [ 305.605441] Lustre: server umount lustre-OST0000 complete [ 308.195321] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 308.197676] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 320.486172] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 320.490498] Lustre: Skipped 9 previous similar messages [ 322.016344] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 322.139433] Lustre: server umount lustre-MDT0000 complete [ 324.533791] LustreError: 15094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790959203 with bad export cookie 1445685136879294304 [ 324.537604] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 324.543285] LustreError: 15094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 324.726190] Lustre: server umount lustre-MDT0001 complete [ 327.908543] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 12:40:06 (1790959206) [ 337.992563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 338.273411] LustreError: 19944:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 338.317784] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 340.621954] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 343.521838] LustreError: 19945:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 345.137751] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 347.659390] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 351.988895] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 355.188762] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 356.343553] Lustre: Failing over lustre-MDT0000 [ 356.481538] Lustre: server umount lustre-MDT0000 complete [ 360.930080] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 360.931744] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 360.934202] LustreError: 19945:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 360.937935] Lustre: Skipped 2 previous similar messages [ 366.049442] LustreError: 19944:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 366.056650] LustreError: 19944:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 371.598435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 371.678807] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 374.389824] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 377.312536] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 377.322839] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 377.346756] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 380.124529] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 386.528725] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 386.534365] Lustre: Skipped 1 previous similar message [ 386.537400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 397.279120] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 397.353602] Lustre: server umount lustre-MDT0000 complete [ 399.390714] LustreError: 19924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790959278 with bad export cookie 1445685136879295718 [ 399.393913] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 399.395606] LustreError: 19924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 399.564324] Lustre: server umount lustre-MDT0001 complete [ 404.867762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 405.165372] LustreError: 23280:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 405.173620] LustreError: 23280:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 405.200451] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 405.204693] Lustre: Skipped 2 previous similar messages [ 407.503575] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 412.273684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 414.668987] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 418.972822] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 421.941373] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 423.286247] Lustre: Failing over lustre-MDT0000 [ 423.413030] Lustre: server umount lustre-MDT0000 complete [ 428.003317] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 428.005908] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 428.010039] LustreError: 23277:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 428.010049] LustreError: 23277:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 428.015970] Lustre: Skipped 1 previous similar message [ 438.065451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 438.146719] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 440.177988] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 443.360665] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 443.363187] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 443.365421] Lustre: Skipped 1 previous similar message [ 443.372219] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 445.008829] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 448.481857] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 453.507590] Lustre: server umount lustre-MDT0000 complete [ 455.228032] LustreError: 23261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790959334 with bad export cookie 1445685136879296656 [ 455.231209] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 455.234244] LustreError: 23261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 455.368866] Lustre: server umount lustre-MDT0001 complete [ 459.121978] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 12:42:17 (1790959337) [ 466.532790] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [ 471.167917] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 489.442756] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 494.752621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 494.866620] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 494.886339] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 494.932465] Lustre: lustre-MDT0000: new disk, initializing [ 494.998663] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 497.175310] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 503.247181] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 503.326830] Lustre: 29639:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 503.508207] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 505.479615] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 511.033188] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 513.651990] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 517.317611] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 517.489657] Lustre: 31606:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 519.481589] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 519.517789] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 519.527328] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 520.930324] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 526.087278] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 530.013087] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 540.132897] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 540.134262] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 540.135269] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 540.135273] Lustre: Skipped 7 previous similar messages [ 540.140233] LustreError: Skipped 1 previous similar message [ 540.152532] Lustre: Skipped 3 previous similar messages [ 543.607892] Lustre: server umount lustre-OST0000 complete [ 551.276033] Lustre: server umount lustre-MDT0000 complete [ 552.945605] LustreError: 29633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790959432 with bad export cookie 1445685136879297062 [ 552.949084] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 552.950166] LustreError: 29633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 553.155600] Lustre: server umount lustre-MDT0001 complete [ 558.076514] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 559.327959] Key type lgssc unregistered [ 559.466392] LNet: 33558:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 559.469729] LNetError: 33558:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 559.479307] LNet: Removed LNI 192.168.202.121@tcp [ 559.872151] Key type .llcrypt unregistered [ 559.874254] Key type ._llcrypt unregistered [ 570.849057] Key type ._llcrypt registered [ 570.850785] Key type .llcrypt registered [ 570.900626] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 571.258694] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 571.327802] alg: No test for adler32 (adler32-zlib) [ 572.222789] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 572.351455] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 573.959214] Key type lgssc registered [ 574.514922] Lustre: Echo OBD driver; http://www.lustre.org/ [ 578.098957] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 578.109587] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 579.342544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 581.228460] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 585.785088] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 585.949935] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 587.701393] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 590.921856] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 593.580239] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 596.619588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 596.769171] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 599.317056] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 602.457355] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 611.116092] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 620.286507] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 12:44:58 (1790959498) [ 621.287537] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 622.273564] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 625.869733] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 12:45:04 (1790959504) [ 630.584359] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 12:45:08 (1790959508) [ 631.134308] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 635.678636] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 635.682406] Lustre: Skipped 2 previous similar messages [ 636.674847] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 12:45:15 (1790959515) [ 641.148256] Lustre: Setting parameter lustre-OST0000-osc.osc.max_pages_per_rpc=1024 in log lustre-client [ 641.151955] Lustre: Skipped 4 previous similar messages [ 649.743950] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 12:45:28 (1790959528) [ 651.239195] Lustre: *** cfs_fail_loc=131b, val=2*** [ 652.958395] Lustre: *** cfs_fail_loc=131b, val=2*** [ 657.969306] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 12:45:36 (1790959536) [ 658.882695] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 658.885667] Lustre: Skipped 9 previous similar messages [ 664.555293] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 12:45:43 (1790959543) [ 670.688357] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 670.691864] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 670.697396] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 672.740168] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 672.740341] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 672.741435] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 675.350737] Lustre: server umount lustre-OST0000 complete [ 677.859229] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 677.867827] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 677.870230] Lustre: Skipped 1 previous similar message [ 682.976733] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 682.983295] Lustre: Skipped 1 previous similar message [ 683.098752] Lustre: server umount lustre-MDT0000 complete [ 684.673118] LustreError: 37904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790959563 with bad export cookie 10880468285811693929 [ 684.675558] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 684.677836] LustreError: 37904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 684.842322] Lustre: server umount lustre-MDT0001 complete [ 689.166136] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 690.426772] Key type lgssc unregistered [ 690.568336] LNet: 42818:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 690.572574] LNetError: 42818:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 690.583506] LNet: Removed LNI 192.168.202.121@tcp [ 690.955905] Key type .llcrypt unregistered [ 690.957700] Key type ._llcrypt unregistered [ 698.697221] Key type ._llcrypt registered [ 698.698525] Key type .llcrypt registered [ 698.742427] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [ 706.772182] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 707.135319] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 707.176029] alg: No test for adler32 (adler32-zlib) [ 708.059944] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 708.203026] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 709.831338] Key type lgssc registered [ 710.351280] Lustre: Echo OBD driver; http://www.lustre.org/ [ 725.361380] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 725.372724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 726.469091] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 726.483659] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 726.523051] Lustre: lustre-MDT0000: new disk, initializing [ 726.563244] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 726.570911] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 728.334434] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 733.523569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 733.570298] Lustre: 45957:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 733.595870] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 733.599564] Lustre: Skipped 1 previous similar message [ 733.643739] Lustre: lustre-MDT0001: new disk, initializing [ 733.714284] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 733.730931] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 733.734581] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 735.459647] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 739.982073] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 742.673944] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 746.069808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 746.234407] Lustre: lustre-OST0000: new disk, initializing [ 746.237186] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 746.241049] Lustre: 47926:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 746.286082] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 748.842385] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 753.642508] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 755.204489] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 755.209700] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 755.257195] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 755.452830] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 755.539716] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 757.309912] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 757.402421] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 760.293035] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 760.293245] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 760.295526] LustreError: Skipped 1 previous similar message [ 760.296413] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 760.301929] Lustre: Skipped 1 previous similar message [ 764.314527] Lustre: server umount lustre-OST0000 complete [ 769.504133] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 769.507766] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 769.516076] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 769.518965] Lustre: Skipped 1 previous similar message [ 770.528286] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 770.529235] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 775.648395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 775.651540] Lustre: Skipped 2 previous similar messages [ 780.255119] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 780.328257] Lustre: server umount lustre-MDT0000 complete [ 781.833283] LustreError: 46756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790959660 with bad export cookie 1136869826791686317 [ 781.836630] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 781.839315] LustreError: 46756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 781.980236] Lustre: server umount lustre-MDT0001 complete [ 785.871158] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 12:47:44 (1790959664) [ 788.706298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 788.880409] LustreError: 50382:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 788.889052] LustreError: 50382:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 788.911120] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 790.547631] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 794.081085] LustreError: 50383:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 794.742211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 794.921342] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 796.625948] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 799.790884] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 802.208201] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 804.955878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 805.100639] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 807.387294] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 810.160455] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 817.659400] LustreError: 52910:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 818.071827] Lustre: Modifying parameter general.timeout=1 in log params [ 819.204081] Lustre: Modifying parameter general.timeout=4 in log params [ 819.206408] Lustre: Skipped 2 previous similar messages [ 821.417064] Lustre: Modifying parameter general.timeout=10 in log params [ 821.419149] Lustre: Skipped 5 previous similar messages [ 825.431968] Lustre: Modifying parameter general.timeout=20 in log params [ 825.434839] Lustre: Skipped 9 previous similar messages [ 833.644535] Lustre: Modifying parameter general.timeout=41 in log params [ 833.647429] Lustre: Skipped 20 previous similar messages [ 849.739834] Lustre: Modifying parameter general.timeout=84 in log params [ 849.746703] Lustre: Skipped 42 previous similar messages [ 869.528473] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 12:49:08 (1790959748) [ 870.646792] Lustre: 59812:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 872.124986] Lustre: 60018:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 872.130652] Lustre: 60018:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 874.326567] Lustre: 60312:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 874.331524] Lustre: 60312:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 881.972462] Lustre: Disabling parameter general.ost.OSS.ost_io.nrs_tbf_rule=start rule3 jobid={cat.0.0} rate=1000 in log params [ 881.975854] Lustre: Skipped 65 previous similar messages [ 884.603784] Lustre: 61661:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 884.612151] LustreError: 61661:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 886.346993] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 12:49:24 (1790959764) [ 892.387956] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 892.393742] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 892.404971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 897.635747] Lustre: server umount lustre-MDT0000 complete [ 899.165991] LustreError: 50363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790959778 with bad export cookie 1136869826791687185 [ 899.167958] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 899.170922] LustreError: 50363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 899.310327] Lustre: server umount lustre-MDT0001 complete [ 910.050920] Lustre: server umount lustre-OST0000 complete [ 921.449520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 921.625491] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 921.771267] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 923.502489] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 927.208382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 927.265396] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 927.279986] Lustre: 64189:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 927.319647] LustreError: 64200:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 929.092539] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 932.175386] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 934.500950] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 937.406683] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 937.501182] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 940.292380] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 943.293135] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 949.423408] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 949.429570] Lustre: Skipped 3 previous similar messages [ 949.447867] Lustre: 66754:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 953.824549] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 953.829123] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 953.834849] LustreError: Skipped 1 previous similar message [ 953.841486] Lustre: Skipped 2 previous similar messages [ 953.844026] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 953.848138] Lustre: Skipped 4 previous similar messages [ 957.805489] Lustre: server umount lustre-OST0000 complete [ 963.553175] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 963.558620] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 965.628254] Lustre: server umount lustre-MDT0000 complete [ 967.552450] LustreError: 64182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790959846 with bad export cookie 1136869826791689614 [ 967.558262] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 967.561263] LustreError: 64182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 967.718175] Lustre: server umount lustre-MDT0001 complete [ 973.456283] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 975.061367] Key type lgssc unregistered [ 975.229594] LNet: 67958:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 975.235667] LNetError: 67958:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 975.245224] LNet: Removed LNI 192.168.202.121@tcp [ 975.770738] Key type .llcrypt unregistered [ 975.772736] Key type ._llcrypt unregistered [ 989.012170] Key type ._llcrypt registered [ 989.013452] Key type .llcrypt registered [ 989.064258] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 989.591783] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 989.604839] alg: No test for adler32 (adler32-zlib) [ 990.506364] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 990.639776] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 992.255233] Key type lgssc registered [ 992.839899] Lustre: Echo OBD driver; http://www.lustre.org/ [ 996.759717] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 996.773440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 998.041272] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 999.873170] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1004.018990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1004.310715] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1006.455719] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1009.802780] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1012.224946] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1015.228907] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1015.547216] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1018.252986] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1021.680799] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1035.143098] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1040.354022] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1040.359196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1040.361353] Lustre: Skipped 2 previous similar messages [ 1040.361683] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1040.369350] Lustre: Skipped 2 previous similar messages [ 1044.453643] Lustre: server umount lustre-MDT0000 complete [ 1045.472769] LustreError: 70094:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1045.480978] LustreError: 70094:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 1045.981265] LustreError: 71215:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790959925 with bad export cookie 15598052741644958523 [ 1045.983970] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1045.986646] LustreError: 71215:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1046.119974] Lustre: server umount lustre-MDT0001 complete [ 1057.065728] Lustre: server umount lustre-OST0000 complete [ 1068.920047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1069.090819] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1069.112127] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1069.242621] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1071.423071] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1075.697198] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1075.748093] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1075.774720] Lustre: 74247:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1075.970074] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1077.947108] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1081.484688] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1084.449832] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1087.847596] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1087.941695] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1088.110024] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1090.622432] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1093.862686] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1099.947351] Lustre: 76821:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1100.759421] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1104.353532] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1104.353795] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1104.354885] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1104.358892] Lustre: Skipped 1 previous similar message [ 1104.361929] LustreError: Skipped 1 previous similar message [ 1108.445072] Lustre: server umount lustre-OST0000 complete [ 1112.033121] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1112.035902] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1112.041293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1112.044267] Lustre: Skipped 1 previous similar message [ 1114.592933] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1114.593307] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1116.316853] Lustre: server umount lustre-MDT0000 complete [ 1117.843755] LustreError: 74240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790959996 with bad export cookie 15598052741644959342 [ 1117.844048] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1117.847316] LustreError: 74240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1117.990077] Lustre: server umount lustre-MDT0001 complete [ 1123.301119] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 1124.727608] Key type lgssc unregistered [ 1124.863764] LNet: 78027:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1124.867648] LNetError: 78027:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1124.875325] LNet: Removed LNI 192.168.202.121@tcp [ 1125.256250] Key type .llcrypt unregistered [ 1125.258091] Key type ._llcrypt unregistered [ 1136.339397] Key type ._llcrypt registered [ 1136.340622] Key type .llcrypt registered [ 1136.400655] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 1136.730801] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1136.744433] alg: No test for adler32 (adler32-zlib) [ 1137.617233] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 1137.717794] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 1139.303214] Key type lgssc registered [ 1139.784468] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1142.782345] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1142.789112] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1144.012654] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1145.929930] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1149.880649] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1150.214442] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1151.948655] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1154.846694] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1157.318373] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1160.427797] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1160.746056] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1163.179977] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1166.710533] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1179.431850] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1180.270014] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1180.273600] Lustre: Skipped 1 previous similar message [ 1181.529975] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1181.533131] Lustre: Skipped 2 previous similar messages [ 1183.552054] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1183.554241] Lustre: Skipped 4 previous similar messages [ 1187.567169] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1187.573997] Lustre: Skipped 8 previous similar messages [ 1195.779343] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1195.782159] Lustre: Skipped 18 previous similar messages [ 1211.974752] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1211.979909] Lustre: Skipped 37 previous similar messages [ 1226.296894] Lustre: 87525:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 1227.623827] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 12:55:06 (1790960106) [ 1228.447792] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 1229.209605] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 12:55:07 (1790960107) [ 1231.329252] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1231.331700] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1231.336579] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1231.344724] Lustre: Skipped 1 previous similar message [ 1236.450716] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1236.453815] Lustre: Skipped 2 previous similar messages [ 1236.696946] Lustre: server umount lustre-OST0000 complete [ 1241.569174] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1241.571975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1241.574282] Lustre: Skipped 1 previous similar message [ 1244.385421] Lustre: server umount lustre-MDT0000 complete [ 1245.927248] LustreError: 79412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790960125 with bad export cookie 1378311682627129971 [ 1245.932298] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1245.933414] LustreError: 79412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1246.054467] Lustre: server umount lustre-MDT0001 complete [ 1250.521205] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 1251.745274] Key type lgssc unregistered [ 1251.901423] LNet: 88773:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1251.904033] LNetError: 88773:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1251.911496] LNet: Removed LNI 192.168.202.121@tcp [ 1252.279277] Key type .llcrypt unregistered [ 1252.280886] Key type ._llcrypt unregistered [ 1254.020628] Key type ._llcrypt registered [ 1254.022139] Key type .llcrypt registered [ 1254.065252] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 1255.663028] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules [ 1255.934496] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1255.949267] alg: No test for adler32 (adler32-zlib) [ 1256.822022] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 1256.924194] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 1258.519193] Key type lgssc registered [ 1258.995142] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1259.506042] LustreError: 89940:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 1267.798905] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 1271.680951] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1281.149210] LustreError: 89940:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 1281.201382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1282.527169] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1286.128161] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1292.775526] LustreError: 90965:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1292.788111] LustreError: 90965:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1293.144210] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 12:56:11 (1790960171) [ 1297.096853] LustreError: 90996:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1297.103627] LustreError: 90996:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1297.116101] LustreError: 90996:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 15, retries 0, failed: rc = -5 [ 1297.336280] Lustre: server umount lustre-MDT0000 complete [ 1308.650832] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 1311.357227] Key type lgssc unregistered [ 1311.595654] LNet: 92488:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1311.609368] LNetError: 92488:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1312.680594] LNet: Removed LNI 192.168.202.121@tcp [ 1313.391667] Key type .llcrypt unregistered [ 1313.395685] Key type ._llcrypt unregistered [ 1326.690107] Key type ._llcrypt registered [ 1326.691855] Key type .llcrypt registered [ 1326.767388] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 1327.656708] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1327.681672] alg: No test for adler32 (adler32-zlib) [ 1328.753719] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 1328.951789] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 1330.599274] Key type lgssc registered [ 1331.649371] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1337.038128] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1337.053319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1338.526089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1341.401718] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1345.977579] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1346.317842] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1348.302325] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1351.835868] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1354.795962] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1358.086287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1358.412733] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1361.818724] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1365.528707] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1399.131555] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 12:57:57 (1790960277) [ 1400.119382] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 1401.071112] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 12:57:59 (1790960279) [ 1402.848722] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1402.850192] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1402.854143] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1402.858270] Lustre: Skipped 2 previous similar messages [ 1402.861220] Lustre: Skipped 1 previous similar message [ 1407.971502] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1407.974402] Lustre: Skipped 3 previous similar messages [ 1408.876491] Lustre: server umount lustre-MDT0000 complete [ 1411.066070] LustreError: 93681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790960290 with bad export cookie 11143643855405904399 [ 1411.073417] LustreError: 93681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1411.074728] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1411.258493] Lustre: server umount lustre-MDT0001 complete [ 1423.661800] Lustre: server umount lustre-OST0000 complete [ 1430.205549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1430.449620] LustreError: 97970:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1430.465703] LustreError: 97970:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1430.591644] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1432.697633] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1435.618947] LustreError: 97970:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1437.181308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1437.555482] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1439.540974] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1442.926518] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1445.526379] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1452.226975] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1452.351762] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1452.360063] LustreError: 97959:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1452.376712] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1452.382695] LustreError: 100053:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -98 [ 1452.386630] LustreError: 100053:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 1452.389146] LustreError: 100053:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 1452.428147] Lustre: server umount lustre-OST0000 complete [ 1452.430315] LustreError: 100053:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1456.897475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1457.015393] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1457.018336] LustreError: 97960:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1457.022435] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1457.026955] LustreError: 100515:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -98 [ 1457.030156] LustreError: 100515:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 1457.032477] LustreError: 100515:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 1457.066957] Lustre: server umount lustre-OST0000 complete [ 1457.069769] LustreError: 100515:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1461.464375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1461.581506] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1461.584827] Lustre: Found index 0 for lustre-OST0000, updating log [ 1461.587655] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 1461.625228] Lustre: lustre-OST0000: new disk, initializing [ 1461.627718] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1461.631507] Lustre: 101044:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1461.797700] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1464.990971] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1467.205140] Lustre: Failing over lustre-OST0000 [ 1467.349887] Lustre: server umount lustre-OST0000 complete [ 1472.992479] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1472.993401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1472.999891] Lustre: Skipped 1 previous similar message [ 1475.635555] Lustre: server umount lustre-MDT0000 complete [ 1477.780561] LustreError: 97951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790960356 with bad export cookie 11143643855405907143 [ 1477.785653] LustreError: 98703:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0001: can't precreate: rc = -5 [ 1477.785790] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1477.788855] LustreError: 97951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1477.802177] LustreError: 98703:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0001: cannot precreate objects: rc = -5 [ 1487.418531] Lustre: server umount lustre-MDT0001 complete [ 1492.121571] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 12:59:30 (1790960370) [ 1499.725611] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 1504.827713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1505.028301] LustreError: 103387:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1505.076618] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1507.051515] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1510.369134] LustreError: 103388:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1511.123275] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1513.551293] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1515.160037] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 1518.559575] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1518.890218] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 1520.935362] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1520.952686] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1520.955871] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1521.604378] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1525.588807] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1525.626198] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 1525.672971] Lustre: lustre-OST0001: new disk, initializing [ 1525.675983] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1525.679823] Lustre: 105710:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1528.473543] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1529.227548] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1529.232626] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1529.302024] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1532.381328] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1538.587077] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1540.577078] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1540.580154] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1540.582339] Lustre: Skipped 1 previous similar message [ 1540.585176] Lustre: Skipped 1 previous similar message [ 1544.960902] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.21@tcp (stopping) [ 1546.539489] Lustre: server umount lustre-MDT0001 complete [ 1550.072050] LustreError: 103383:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1550.234901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1550.276948] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1550.279439] Lustre: Found index 1 for lustre-MDT0001, updating log [ 1550.296285] Lustre: 103376:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1550.578177] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1550.581420] Lustre: Skipped 3 previous similar messages [ 1551.718928] LustreError: 107375:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 1551.723913] LustreError: 107375:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 1551.727847] LustreError: 103379:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 1551.773826] LustreError: 107397:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.121@tcp: cfg command failed: rc = -17 [ 1551.776470] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 1551.776470] [ 1552.220830] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1555.944225] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1556.394392] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 13:00:35 (1790960435) [ 1563.143482] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 1570.080518] Lustre: 109318:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1579.800505] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1581.134650] Lustre: 110450:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1632.223873] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1632.228407] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1632.233794] Lustre: Skipped 1 previous similar message [ 1632.236557] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1632.238812] Lustre: Skipped 3 previous similar messages [ 1638.294914] Lustre: server umount lustre-MDT0000 complete [ 1639.716880] LustreError: 103368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790960518 with bad export cookie 11143643855405907983 [ 1639.718497] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1639.721687] LustreError: 103368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1639.830637] Lustre: server umount lustre-MDT0001 complete [ 1651.213427] Lustre: server umount lustre-OST0000 complete [ 1662.905067] Lustre: server umount lustre-OST0001 complete [ 1665.126593] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1668.624331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1671.736792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1675.260039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1682.383814] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 1686.184832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1686.193790] Lustre: lustre-MDT0000: reset Object Index mappings [ 1686.367234] LustreError: 114412:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1686.400076] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1687.848653] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1690.818448] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1690.826911] Lustre: lustre-MDT0001: reset Object Index mappings [ 1692.482211] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1693.483384] Lustre: 115559:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1695.730768] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1697.000653] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:350 to 0x280000401:386) [ 1697.961457] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1700.915923] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1701.096721] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:199 to 0x280000400:225) [ 1703.256628] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1706.474912] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 1706.476646] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 1706.633758] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1707.849987] Lustre: 117432:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1722.444730] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 13:03:21 (1790960601) [ 1752.547606] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1752.548929] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1752.549300] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1752.549307] Lustre: Skipped 7 previous similar messages [ 1752.552901] Lustre: Skipped 6 previous similar messages [ 1757.861407] Lustre: server umount lustre-MDT0000 complete [ 1759.303718] LustreError: 117682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790960638 with bad export cookie 11143643855405954785 [ 1759.309534] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1762.784160] LustreError: 115939:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1762.784252] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1762.791042] LustreError: 115939:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1762.796592] Lustre: Skipped 1 previous similar message [ 1765.637985] Lustre: server umount lustre-MDT0001 complete [ 1773.188402] Lustre: server umount lustre-OST0000 complete [ 1780.573509] Lustre: server umount lustre-OST0001 complete [ 1782.719269] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [ 1786.326555] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 1798.949202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1799.030422] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1799.041074] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1799.077521] Lustre: lustre-MDT0000: new disk, initializing [ 1799.102186] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1800.449196] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1803.397285] Lustre: server umount lustre-MDT0000 complete [ 1806.515328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1806.612828] Lustre: Found index 0 for lustre-MDT0000, updating log [ 1806.618227] Lustre: 121688:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1806.624160] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 1808.045395] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1809.799162] Lustre: server umount lustre-MDT0000 complete [ 1812.384111] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 13:04:51 (1790960691) [ 1812.878873] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 1813.402515] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 13:04:52 (1790960692) [ 1819.034297] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 1822.504718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1822.637832] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1822.640622] Lustre: Skipped 5 previous similar messages [ 1824.098682] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1827.020679] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1827.071881] Lustre: 123736:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1827.095307] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1827.097218] Lustre: Skipped 1 previous similar message [ 1827.133597] Lustre: lustre-MDT0001: new disk, initializing [ 1827.185561] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1827.190579] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1828.772029] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1831.106744] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 1833.654228] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1833.760172] Lustre: lustre-OST0000: new disk, initializing [ 1833.762162] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1833.764513] Lustre: 125312:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1835.064669] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1835.069175] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1835.103888] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1835.761076] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1839.883562] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1839.952617] Lustre: 126184:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1841.346505] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1841.349296] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1841.385286] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1841.910457] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 1846.291060] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1847.464593] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2113.273969] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 13:09:51 (1790960991) [ 2125.798429] Lustre: server umount lustre-MDT0000 complete [ 2128.172312] LustreError: 123728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790961007 with bad export cookie 11143643855406008090 [ 2128.189058] LustreError: 123728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2128.191127] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2128.357786] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2128.362655] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2128.363386] LustreError: 126978:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2128.363394] LustreError: 126978:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2128.376056] Lustre: Skipped 3 previous similar messages [ 2128.405655] Lustre: Skipped 12 previous similar messages [ 2134.474789] Lustre: server umount lustre-MDT0001 complete [ 2142.671424] Lustre: server umount lustre-OST0000 complete [ 2150.712277] Lustre: server umount lustre-OST0001 complete [ 2154.129569] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [ 2159.695850] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 2179.291650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2179.473415] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2179.500754] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2179.503052] Lustre: Skipped 1 previous similar message [ 2179.541739] Lustre: lustre-MDT0000: new disk, initializing [ 2179.545176] Lustre: Skipped 1 previous similar message [ 2179.603959] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2179.609505] Lustre: Skipped 3 previous similar messages [ 2179.622207] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2182.031608] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2188.917965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2189.041404] Lustre: 135575:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2189.248412] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2191.686862] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2197.210934] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2200.843192] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2205.489429] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2205.703483] Lustre: lustre-OST0000: new disk, initializing [ 2205.705583] Lustre: Skipped 1 previous similar message [ 2205.708139] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2205.710291] Lustre: Skipped 2 previous similar messages [ 2205.713826] Lustre: 137541:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2207.694934] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2207.702222] Lustre: Skipped 1 previous similar message [ 2207.705592] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2207.778137] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2209.578921] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2215.352470] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2220.732615] Lustre: lustre-MDD0000: changelog on [ 2222.625488] Lustre: lustre-MDD0001: changelog on [ 2840.429973] Lustre: lustre-MDD0001: changelog off [ 2841.083473] Lustre: 137715:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 2841.404770] Lustre: lustre-MDD0000: changelog off [ 2847.200065] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2847.203370] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2847.208047] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2847.210546] Lustre: Skipped 2 previous similar messages [ 2847.713770] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2849.049197] Lustre: server umount lustre-OST0000 complete [ 2852.832843] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2852.837788] Lustre: Skipped 2 previous similar messages [ 2856.508415] Lustre: server umount lustre-MDT0000 complete [ 2857.887461] LustreError: 135566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790961737 with bad export cookie 11143643855406770173 [ 2857.889046] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2857.893026] LustreError: 135566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2858.016297] Lustre: server umount lustre-MDT0001 complete [ 2861.975445] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 2863.034770] Key type lgssc unregistered [ 2863.173384] LNet: 146230:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2863.176628] LNetError: 146230:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2863.186306] LNet: Removed LNI 192.168.202.121@tcp [ 2863.526125] Key type .llcrypt unregistered [ 2863.527774] Key type ._llcrypt unregistered [ 2870.619991] Key type ._llcrypt registered [ 2870.621691] Key type .llcrypt registered [ 2870.667804] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [ 2877.329731] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 2877.767397] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2877.775761] alg: No test for adler32 (adler32-zlib) [ 2878.640867] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 2878.731455] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 2880.319188] Key type lgssc registered [ 2880.729134] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2892.453632] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2892.461963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2893.541920] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2893.550876] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2893.585916] Lustre: lustre-MDT0000: new disk, initializing [ 2893.607676] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2893.613579] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2894.961381] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2899.113911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2899.155917] Lustre: 149416:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2899.178317] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2899.180364] Lustre: Skipped 1 previous similar message [ 2899.211626] Lustre: lustre-MDT0001: new disk, initializing [ 2899.249201] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2899.262988] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2899.266374] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2900.615070] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2904.348430] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2906.110549] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2908.270967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2908.371252] Lustre: lustre-OST0000: new disk, initializing [ 2908.373523] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2908.376936] Lustre: 151386:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2908.405901] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2910.341522] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2914.178407] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2915.855152] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2915.862874] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2915.900414] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2922.905787] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_module obdecho/obdecho [ 2923.674452] LustreError: 152390:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 2923.677227] LustreError: 152390:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 2924.300489] LustreError: 152486:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 2930.144188] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2930.146814] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2930.152590] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2931.168953] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2931.169114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2931.172361] Lustre: Skipped 1 previous similar message [ 2931.174029] Lustre: Skipped 1 previous similar message [ 2933.917751] Lustre: server umount lustre-MDT0000 complete [ 2935.254405] LustreError: 149409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790961814 with bad export cookie 7326966294842105801 [ 2935.258378] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2935.258914] LustreError: 149409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2935.368797] Lustre: server umount lustre-MDT0001 complete [ 2945.944248] Lustre: server umount lustre-OST0000 complete [ 2952.711776] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [ 2955.978779] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 2967.807185] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2967.890291] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2967.901855] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2967.940662] Lustre: lustre-MDT0000: new disk, initializing [ 2967.967176] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2967.974670] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2969.414277] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2973.575263] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2973.620018] Lustre: 156203:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2973.644715] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2973.646259] Lustre: Skipped 1 previous similar message [ 2973.682331] Lustre: lustre-MDT0001: new disk, initializing [ 2973.710928] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2973.724037] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2973.727351] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2975.092893] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2978.819201] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2980.597167] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2982.825667] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2982.922394] Lustre: lustre-OST0000: new disk, initializing [ 2982.925701] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2982.929170] Lustre: 158171:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2982.961841] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2984.947678] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 2986.622129] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2986.625487] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2986.670139] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2988.710154] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2990.158775] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2990.236677] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2991.609922] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2991.685145] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2997.218187] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2997.219034] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2997.219641] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2997.219645] Lustre: Skipped 1 previous similar message [ 2997.220589] LustreError: Skipped 1 previous similar message [ 2997.225795] Lustre: Skipped 1 previous similar message [ 2998.418815] Lustre: server umount lustre-OST0000 complete [ 3002.337103] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3002.337995] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3002.340864] Lustre: Skipped 1 previous similar message [ 3002.342379] Lustre: Skipped 1 previous similar message [ 3005.781845] Lustre: server umount lustre-MDT0000 complete [ 3007.017194] LustreError: 156194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790961886 with bad export cookie 7326966294842106760 [ 3007.020467] LustreError: 156194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3007.020556] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3007.143846] Lustre: server umount lustre-MDT0001 complete [ 3008.307752] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 13:24:47 (1790961887) [ 3008.791326] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 3009.312413] Lustre: DEBUG MARKER: == conf-sanity test 138: MDT mount failure must not LBUG destroying recovery exports ========================================================== 13:24:48 (1790961888) [ 3014.793293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3014.945087] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3016.157761] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3018.704168] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3020.054595] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3022.265696] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3023.907872] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3025.969078] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3027.794752] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3029.985610] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3050.151540] Lustre: *** cfs_fail_loc=718, val=0*** [ 3050.762788] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.21@tcp (stopping) [ 3050.766499] Lustre: Skipped 1 previous similar message [ 3052.005551] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3052.019322] Lustre: Skipped 2 previous similar messages [ 3052.362318] Lustre: server umount lustre-MDT0000 complete [ 3054.914228] LustreError: 161117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790961934 with bad export cookie 7326966294842107628 [ 3054.915716] Lustre: *** cfs_fail_loc=718, val=0*** [ 3054.918783] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3054.921561] LustreError: 161117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3054.928555] Lustre: Skipped 3 previous similar messages [ 3055.091426] Lustre: server umount lustre-MDT0001 complete [ 3060.635921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3065.251617] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x65ae9d067552e9e7 [ 3065.272022] Lustre: MGC192.168.202.121@tcp: Connection restored to 0@lo (at 0@lo) [ 3065.429828] Lustre: *** cfs_fail_loc=723, val=0*** [ 3065.431894] Lustre: Skipped 1 previous similar message [ 3065.494048] LustreError: 164507:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-122) [ 3065.499296] LustreError: 164507:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.121@tcp: cfg command failed: rc = -122 [ 3065.512486] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 3065.512486] [ 3065.519715] LustreError: MGC192.168.202.121@tcp: Confguration from log lustre-MDT0000 failed from MGS -122. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3065.528074] LustreError: 164477:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -122 [ 3065.532452] LustreError: 164477:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -122 [ 3065.538832] LustreError: 164477:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 3065.541821] LustreError: 164477:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 3065.627870] Lustre: server umount lustre-MDT0000 complete [ 3065.636814] LustreError: 164477:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -122 [ 3070.221150] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3070.300666] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3070.433755] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3070.445723] LustreError: 164916:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3070.457526] LustreError: 164916:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3070.544047] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3070.548201] Lustre: Skipped 2 previous similar messages [ 3071.282967] LustreError: 164916:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3071.297642] LustreError: 164916:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3072.610964] LustreError: 162964:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3072.619502] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3072.623104] LustreError: 162964:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3073.311761] Lustre: 147636:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790961936/real 1790961936] req@0000000074acb556 x1877959515396480/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790961952 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3074.103127] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3075.423254] Lustre: 147636:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790961938/real 1790961938] req@000000004db46319 x1877959515396608/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790961954 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3076.347586] LustreError: 164913:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3078.687717] Lustre: 147637:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790961941/real 1790961941] req@0000000087ca078b x1877959515396736/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790961957 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3080.382283] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3080.622620] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 3081.479059] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3081.759253] Lustre: 147637:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790961944/real 1790961944] req@000000004508bcc4 x1877959515396992/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790961960 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3083.785850] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3085.813265] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3085.819892] Lustre: lustre-MDT0001: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 3088.765371] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3089.747675] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3095.960109] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3097.592611] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3111.400101] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3111.403263] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3111.417951] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3111.417958] Lustre: Skipped 3 previous similar messages [ 3111.440257] LustreError: Skipped 1 previous similar message [ 3113.431690] Lustre: server umount lustre-OST0000 complete [ 3131.359171] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3131.529146] Lustre: server umount lustre-MDT0000 complete [ 3131.876657] LustreError: 164915:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3131.886830] LustreError: 164915:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 3133.556785] LustreError: 161870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790962012 with bad export cookie 7326966294842108531 [ 3133.565883] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3133.575939] LustreError: 161870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3133.889096] Lustre: server umount lustre-MDT0001 complete [ 3140.518861] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 3142.427456] Key type lgssc unregistered [ 3142.652302] LNet: 167733:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3142.658556] LNetError: 167733:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3142.672732] LNet: Removed LNI 192.168.202.121@tcp [ 3143.155083] Key type .llcrypt unregistered [ 3143.156825] Key type ._llcrypt unregistered [ 3157.360032] Key type ._llcrypt registered [ 3157.363387] Key type .llcrypt registered [ 3157.455334] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 3158.341897] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3158.422394] alg: No test for adler32 (adler32-zlib) [ 3159.407530] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 3159.619733] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 3161.271286] Key type lgssc registered [ 3162.409977] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3169.108303] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 3169.127541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3170.435366] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3173.715706] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3179.764700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3180.044381] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3183.240723] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3188.303636] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3192.114948] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3196.909318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3197.198184] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3201.176448] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3205.795928] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3216.690612] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3222.823545] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.21@tcp (stopping) [ 3226.081137] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3226.094960] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3226.105109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3227.109589] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3227.120806] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3227.123963] Lustre: Skipped 2 previous similar messages [ 3228.339985] Lustre: server umount lustre-MDT0000 complete [ 3230.978230] LustreError: 169022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790962110 with bad export cookie 8369403484158929739 [ 3230.983665] LustreError: 169022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3230.984068] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3232.225860] LustreError: 169848:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3232.229685] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3232.241936] LustreError: 169848:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3232.255989] Lustre: Skipped 1 previous similar message [ 3232.267324] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3232.275584] Lustre: Skipped 1 previous similar message [ 3233.018786] LustreError: 169036:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3235.236572] LustreError: 169037:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3237.279354] Lustre: server umount lustre-MDT0001 complete [ 3241.655274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3250.343775] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3250.528625] LustreError: 172903:0:(llog.c:1655:llog_backup()) MGC192.168.202.121@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 3250.532928] Lustre: 172903:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.121@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 3253.245143] LustreError: 172908:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3259.385401] LustreError: 172908:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3259.393654] LustreError: 172908:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3269.883660] LustreError: 172909:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3269.903727] LustreError: 172909:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [ 3286.496898] LustreError: 172926:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3286.519661] LustreError: 172926:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 15 previous similar messages [ 3311.583397] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 3311.587409] Lustre: 172903:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.121@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 3319.040760] LustreError: 172909:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3319.062922] LustreError: 172909:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 19 previous similar messages [ 3352.106559] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3354.546471] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3356.141871] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3360.006550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3376.858080] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3376.905820] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 3380.959900] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3382.250859] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3386.074683] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3387.561609] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3392.229417] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3393.244741] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3407.842272] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3407.844383] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3407.848963] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3407.848972] Lustre: Skipped 2 previous similar messages [ 3407.876288] Lustre: Skipped 1 previous similar message [ 3411.979301] Lustre: server umount lustre-MDT0000 complete [ 3412.962419] LustreError: 172908:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3412.981301] LustreError: 172908:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 45 previous similar messages [ 3414.600459] LustreError: 169023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790962293 with bad export cookie 8369403484158936417 [ 3414.601130] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3414.606329] LustreError: 169023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3414.882167] Lustre: server umount lustre-MDT0001 complete [ 3427.714155] Lustre: server umount lustre-OST0000 complete [ 3441.038416] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [ 3447.334133] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 3468.500648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3468.635724] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3468.655849] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3468.718230] Lustre: lustre-MDT0000: new disk, initializing [ 3468.772958] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3468.786283] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3471.774640] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3479.619537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3479.711906] Lustre: 178005:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3479.755387] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3479.758286] Lustre: Skipped 1 previous similar message [ 3479.822880] Lustre: lustre-MDT0001: new disk, initializing [ 3479.939537] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3479.947992] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3483.270978] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3490.449598] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3494.003450] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3498.515832] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3498.698520] Lustre: lustre-OST0000: new disk, initializing [ 3498.700960] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3498.704536] Lustre: 179968:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3498.778298] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3498.782233] Lustre: Skipped 1 previous similar message [ 3502.608340] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3505.689320] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3505.695123] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3505.749575] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3508.273092] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3511.290963] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3511.454954] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3514.756962] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3514.910902] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3520.998492] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3521.002170] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3521.016678] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3521.017310] Lustre: Skipped 2 previous similar messages [ 3521.030145] Lustre: Skipped 3 previous similar messages [ 3522.785395] Lustre: server umount lustre-OST0000 complete [ 3526.124447] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3526.132162] LustreError: Skipped 1 previous similar message [ 3531.996562] Lustre: server umount lustre-MDT0000 complete [ 3534.319619] LustreError: 177997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790962413 with bad export cookie 8369403484158953469 [ 3534.322093] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3534.327310] LustreError: 177997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3534.516814] Lustre: server umount lustre-MDT0001 complete [ 3539.968916] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 13:33:38 (1790962418) [ 3544.537321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3544.848585] LustreError: 182380:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3544.895145] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3547.347979] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3553.027477] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3556.185473] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3561.134823] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3564.556639] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3568.598785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3572.290910] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3576.898994] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3587.281675] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 13:34:25 (1790962465) [ 3590.626773] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3590.636802] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3590.647798] Lustre: Skipped 2 previous similar messages [ 3590.651553] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3590.659306] Lustre: Skipped 4 previous similar messages [ 3595.794660] Lustre: server umount lustre-OST0000 complete [ 3604.562161] Lustre: server umount lustre-MDT0000 complete [ 3606.870865] LustreError: 182362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790962485 with bad export cookie 8369403484158954351 [ 3606.874109] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3606.884126] LustreError: 182362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3607.094779] Lustre: server umount lustre-MDT0001 complete [ 3613.970834] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 3615.780098] Key type lgssc unregistered [ 3615.948032] LNet: 186104:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3615.956478] LNetError: 186104:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3615.968063] LNet: Removed LNI 192.168.202.121@tcp [ 3616.448144] Key type .llcrypt unregistered [ 3616.449919] Key type ._llcrypt unregistered [ 3627.929036] Key type ._llcrypt registered [ 3627.930858] Key type .llcrypt registered [ 3627.983792] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 3628.454414] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3628.468094] alg: No test for adler32 (adler32-zlib) [ 3629.450977] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 3629.628882] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 3631.231213] Key type lgssc registered [ 3631.892770] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3636.792402] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 3636.808990] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3652.319568] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3657.439292] LustreError: 187341:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.121@tcp: failed processing log, type 4: rc = -110 [ 3662.559479] LustreError: 187341:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 3662.569503] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 3662.575214] LustreError: MGC192.168.202.121@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 3662.581836] LustreError: 187341:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-OST0000: -22 [ 3662.585650] LustreError: 187341:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -22 [ 3662.592989] LustreError: 187341:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 3662.625985] Lustre: server umount lustre-OST0000 complete [ 3662.628300] LustreError: 187341:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 3668.191949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3669.530993] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3672.515561] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3678.519064] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3678.733020] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3682.141390] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3686.951199] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3690.288308] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3694.734700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3694.885761] LustreError: 189653:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 3694.893202] Lustre: 189653:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.202.121@tcp: invalid local config log lustre-OST0000: rc = -22 [ 3694.974640] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3699.487899] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3704.471826] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3705.762881] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3705.764210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3705.781023] Lustre: Skipped 2 previous similar messages [ 3710.944084] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3710.953284] Lustre: Skipped 4 previous similar messages [ 3716.071210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3716.077262] Lustre: Skipped 2 previous similar messages [ 3720.161596] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3720.248552] Lustre: server umount lustre-MDT0000 complete [ 3721.186144] LustreError: 187840:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3721.199547] LustreError: 187840:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 3722.520693] LustreError: 187821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790962601 with bad export cookie 16384275928035646263 [ 3722.523215] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3722.533864] LustreError: 187821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3722.715260] Lustre: server umount lustre-MDT0001 complete [ 3735.626977] Lustre: server umount lustre-OST0000 complete [ 3740.666762] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [ 3746.612639] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 3768.098800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3768.259765] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3768.287451] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3768.340931] Lustre: lustre-MDT0000: new disk, initializing [ 3768.391136] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3768.401827] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3771.055714] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3779.068323] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3779.185668] Lustre: 193173:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3779.229268] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3779.232632] Lustre: Skipped 1 previous similar message [ 3779.281650] Lustre: lustre-MDT0001: new disk, initializing [ 3779.347820] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3779.381073] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3779.388365] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3782.109297] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3788.394950] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3792.136836] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3796.255264] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3796.475784] Lustre: lustre-OST0000: new disk, initializing [ 3796.483122] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3796.487460] Lustre: 195137:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3796.564363] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3800.295514] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3802.139829] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3802.146634] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3802.185532] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3806.017732] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3808.377627] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3808.486000] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3810.730840] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3810.891062] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3812.324717] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3812.325553] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3812.333155] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3812.348440] Lustre: Skipped 1 previous similar message [ 3817.442243] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3817.450284] Lustre: Skipped 2 previous similar messages [ 3818.174560] Lustre: server umount lustre-OST0000 complete [ 3820.512927] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3820.521964] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3822.564380] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3827.686204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3827.689307] Lustre: Skipped 4 previous similar messages [ 3834.335177] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3834.460796] Lustre: server umount lustre-MDT0000 complete [ 3836.369499] LustreError: 193966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790962715 with bad export cookie 16384275928035646977 [ 3836.372537] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3836.375594] LustreError: 193966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3836.545578] Lustre: server umount lustre-MDT0001 complete [ 3840.589237] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 13:38:39 (1790962719) [ 3850.376747] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 3851.826170] Key type lgssc unregistered [ 3851.998404] LNet: 198082:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3852.006683] LNetError: 198082:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3852.019521] LNet: Removed LNI 192.168.202.121@tcp [ 3852.496701] Key type .llcrypt unregistered [ 3852.498077] Key type ._llcrypt unregistered [ 3862.748877] Key type ._llcrypt registered [ 3862.750749] Key type .llcrypt registered [ 3862.805660] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 3863.313535] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3863.375074] alg: No test for adler32 (adler32-zlib) [ 3864.324297] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 3864.509792] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 3866.121936] Key type lgssc registered [ 3866.768740] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3871.257527] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 3871.272888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3872.506166] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3875.316547] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3880.419222] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3880.645012] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3882.885285] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3886.564978] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3889.883414] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3894.419957] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3894.580275] Lustre: *** cfs_fail_loc=131e, val=0*** [ 3894.582652] LustreError: 201154:0:(llog.c:1696:llog_backup()) MGC192.168.202.121@tcp: failed to backup log lustre-OST0000: rc = -28 [ 3894.587518] Lustre: 201154:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.202.121@tcp: can't backup local config lustre-OST0000: rc = -28 [ 3894.670291] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3897.875443] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3901.651121] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3902.944496] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3902.945875] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3902.949928] Lustre: Skipped 2 previous similar messages [ 3902.955182] Lustre: Skipped 2 previous similar messages [ 3904.994151] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3910.113196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3910.116266] Lustre: Skipped 4 previous similar messages [ 3915.232891] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3915.237409] Lustre: Skipped 2 previous similar messages [ 3917.279490] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3917.342804] Lustre: server umount lustre-MDT0000 complete [ 3919.093291] LustreError: 199272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790962798 with bad export cookie 5496770064718554019 [ 3919.096492] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3919.099020] LustreError: 199272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3919.351285] LustreError: 200021:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000060476e5d x1877960549227264/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3919.418190] Lustre: server umount lustre-MDT0001 complete [ 3931.598347] Lustre: server umount lustre-OST0000 complete [ 3935.695909] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [ 3940.374051] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 3957.324655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3957.414209] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3957.429823] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3957.468374] Lustre: lustre-MDT0000: new disk, initializing [ 3957.512198] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3957.519269] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3959.176739] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3964.120826] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3964.166939] Lustre: 204674:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3964.195409] Lustre: *** cfs_fail_loc=131e, val=0*** [ 3964.197211] Lustre: Skipped 1 previous similar message [ 3964.199801] LustreError: 205455:0:(llog.c:1696:llog_backup()) MGC192.168.202.121@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 3964.204754] LustreError: 205455:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 3964.207862] Lustre: 205455:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.121@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 3964.217654] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3964.220078] Lustre: Skipped 1 previous similar message [ 3964.257463] Lustre: lustre-MDT0001: new disk, initializing [ 3964.299971] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3964.318656] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3964.323963] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3966.103916] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3971.403134] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3974.191464] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3977.462259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3977.601803] Lustre: *** cfs_fail_loc=131e, val=0*** [ 3977.603717] Lustre: Skipped 1 previous similar message [ 3977.605644] LustreError: 206618:0:(llog.c:1696:llog_backup()) MGC192.168.202.121@tcp: failed to backup log lustre-OST0000: rc = -28 [ 3977.608949] LustreError: 206618:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 3977.611765] Lustre: 206618:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.121@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 3977.615221] Lustre: 206618:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 3977.627430] Lustre: lustre-OST0000: new disk, initializing [ 3977.630540] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3977.634571] Lustre: 206642:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3977.672341] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3980.290505] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 3984.397395] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3984.406502] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3984.453960] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3984.978919] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3987.264324] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3987.410299] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3989.787392] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3989.887946] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3994.594319] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3994.608793] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3994.612092] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3994.619864] Lustre: Skipped 1 previous similar message [ 4004.834355] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4004.839588] Lustre: Skipped 3 previous similar messages [ 4005.344195] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4005.425896] Lustre: server umount lustre-OST0000 complete [ 4009.955681] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4009.961152] Lustre: Skipped 2 previous similar messages [ 4013.360486] Lustre: server umount lustre-MDT0000 complete [ 4015.075299] LustreError: 205837:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4015.084900] LustreError: 205837:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4015.104280] LustreError: 205468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790962894 with bad export cookie 5496770064718554712 [ 4015.105447] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4015.116959] LustreError: 205468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4015.281531] Lustre: server umount lustre-MDT0001 complete [ 4019.862754] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 13:41:38 (1790962898) [ 4028.180306] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 4032.736327] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4032.947699] LustreError: 209764:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4032.969931] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4034.888683] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4038.112297] LustreError: 209765:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4038.941194] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4041.376814] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4042.743486] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4045.676430] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4048.501913] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4053.229142] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4053.324671] Lustre: lustre-OST0001: new disk, initializing [ 4053.326845] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4053.330919] Lustre: 212070:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4056.143975] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4056.148784] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4056.185577] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4056.272719] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4062.776469] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4068.163126] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4069.064977] Lustre: DEBUG MARKER: ADD OST3 [ 4071.601111] loop: module loaded [ 4071.622516] loop0: detected capacity change from 0 to 204800000 [ 4071.728787] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4071.732359] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4071.740782] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4074.894272] loop0: detected capacity change from 0 to 204800000 [ 4077.310640] Lustre: DEBUG MARKER: STOP OST3 [ 4078.163385] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4078.257860] Lustre: lustre-OST0003: new disk, initializing [ 4078.260310] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 4078.264038] Lustre: 213923:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4078.340617] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 4078.345337] Lustre: Skipped 3 previous similar messages [ 4079.748469] Lustre: *** cfs_fail_loc=2109, val=2*** [ 4079.810621] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 4079.817628] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 4079.840929] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 4080.658767] Lustre: Failing over lustre-OST0003 [ 4080.731842] Lustre: server umount lustre-OST0003 complete [ 4081.761491] LustreError: 214142:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 4082.078908] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4084.505715] LustreError: 211251:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.202.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4085.222664] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4085.222908] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4085.227195] Lustre: Skipped 1 previous similar message [ 4086.724254] Lustre: DEBUG MARKER: START OST3 again [ 4089.599571] LustreError: 211250:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.202.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4089.614231] LustreError: 211250:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4090.432406] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4090.576652] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 4091.849340] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4092.470770] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4092.471430] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4093.703073] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4108.490913] Lustre: Failing over lustre-OST0003 [ 4108.539934] Lustre: server umount lustre-OST0003 complete [ 4111.337906] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4111.339284] LustreError: 216029:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4111.344308] Lustre: Skipped 1 previous similar message [ 4111.354931] LustreError: 216029:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4121.569960] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4121.571726] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4121.575327] Lustre: Skipped 2 previous similar messages [ 4121.580349] Lustre: Skipped 4 previous similar messages [ 4126.176222] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4127.660308] Lustre: server umount lustre-MDT0000 complete [ 4129.140465] LustreError: 209745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790963008 with bad export cookie 5496770064718555580 [ 4129.142366] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4129.146454] LustreError: 209745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4131.808837] LustreError: 209760:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4131.815388] LustreError: 209760:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 4135.425372] Lustre: server umount lustre-MDT0001 complete [ 4137.176665] Lustre: server umount lustre-OST0000 complete [ 4138.871879] Lustre: server umount lustre-OST0001 complete [ 4141.498080] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [ 4144.944948] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 4160.251549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4160.360230] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4160.378981] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4160.417592] Lustre: lustre-MDT0000: new disk, initializing [ 4160.448066] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4160.450383] Lustre: Skipped 1 previous similar message [ 4160.456162] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4162.375553] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4167.313815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4167.360897] Lustre: 219362:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4167.485263] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4169.247989] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4173.717345] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4176.248117] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4179.246881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4179.412663] Lustre: lustre-OST0000: new disk, initializing [ 4179.414442] Lustre: Skipped 1 previous similar message [ 4179.416940] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4179.420056] Lustre: Skipped 2 previous similar messages [ 4179.425498] Lustre: 221332:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4181.207662] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4181.215991] Lustre: Skipped 1 previous similar message [ 4181.222340] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4181.259397] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4182.267840] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4187.077190] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4188.930723] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4189.015230] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4190.828918] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4190.941460] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4196.832821] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4196.835502] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4196.845323] LustreError: Skipped 1 previous similar message [ 4196.852849] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4196.861758] Lustre: Skipped 4 previous similar messages [ 4196.882665] Lustre: Skipped 8 previous similar messages [ 4198.351845] Lustre: server umount lustre-OST0000 complete [ 4207.498646] Lustre: server umount lustre-MDT0000 complete [ 4211.207743] LustreError: 219356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790963090 with bad export cookie 5496770064718557848 [ 4211.216809] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4211.226890] LustreError: 219356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4211.531654] Lustre: server umount lustre-MDT0001 complete [ 4214.918220] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 13:44:52 (1790963092) [ 4235.385798] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [ 4250.954888] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 4319.325903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4319.770858] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4319.812334] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4319.916868] Lustre: lustre-MDT0000: new disk, initializing [ 4320.019752] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4320.025718] Lustre: Skipped 2 previous similar messages [ 4320.039860] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4327.098820] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4339.726824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4339.839736] Lustre: 225948:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4340.243591] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4344.133715] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4353.826489] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4359.900383] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4367.424489] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4367.714943] Lustre: 227918:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4369.780971] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4375.786110] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4386.813749] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4392.846912] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4393.053832] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4398.185908] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4398.481730] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4400.610617] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4400.622879] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4400.629639] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4400.629647] Lustre: Skipped 5 previous similar messages [ 4400.646265] Lustre: Skipped 3 previous similar messages [ 4407.082209] Lustre: server umount lustre-OST0000 complete [ 4411.873474] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4411.884625] LustreError: Skipped 1 previous similar message [ 4418.005104] Lustre: server umount lustre-MDT0000 complete [ 4421.105670] LustreError: 227277:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4421.143789] LustreError: 227277:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4422.254652] LustreError: 225940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790963301 with bad export cookie 5496770064718558730 [ 4422.261069] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4422.270309] LustreError: 225940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4422.821585] Lustre: server umount lustre-MDT0001 complete [ 4432.170445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4438.288991] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4448.886299] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4456.330357] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4467.894344] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4476.023918] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4484.572911] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4493.128620] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4502.578610] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4514.785988] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4514.788703] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4514.808476] Lustre: Skipped 2 previous similar messages [ 4519.677627] Lustre: server umount lustre-OST0000 complete [ 4530.126377] Lustre: server umount lustre-MDT0000 complete [ 4533.219671] LustreError: 230862:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4533.240527] LustreError: 230862:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 4534.674137] LustreError: 230114:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790963413 with bad export cookie 5496770064718559612 [ 4534.693273] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4534.694308] LustreError: 230114:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4535.386792] Lustre: server umount lustre-MDT0001 complete [ 4642.813651] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 4645.688733] Key type lgssc unregistered [ 4645.941702] LNet: 233799:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4645.954591] LNetError: 233799:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4645.987219] LNet: Removed LNI 192.168.202.121@tcp [ 4647.012153] Key type .llcrypt unregistered [ 4647.014046] Key type ._llcrypt unregistered [ 4671.515588] Key type ._llcrypt registered [ 4671.520495] Key type .llcrypt registered [ 4671.608191] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [ 4687.899454] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 4688.518160] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4688.587297] alg: No test for adler32 (adler32-zlib) [ 4690.061218] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 4690.297351] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 4691.993203] Key type lgssc registered [ 4693.213805] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4726.925151] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 4726.971731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4728.280497] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4728.320278] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4728.380823] Lustre: lustre-MDT0000: new disk, initializing [ 4728.455195] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4728.474165] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4732.280534] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4741.577448] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4741.693547] Lustre: 237179:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4741.739925] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4741.743805] Lustre: Skipped 1 previous similar message [ 4741.801946] Lustre: lustre-MDT0001: new disk, initializing [ 4741.873203] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4741.912464] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4741.925686] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4745.645252] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4754.924309] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4764.922048] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4776.160679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4776.368573] Lustre: lustre-OST0000: new disk, initializing [ 4776.375030] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4776.381656] Lustre: 239149:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4776.467089] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4783.266362] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4783.288715] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4783.505820] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4785.857280] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4796.680189] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4801.861635] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4802.094965] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4806.928730] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4807.193711] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4809.188673] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4809.202869] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4809.205097] Lustre: Skipped 1 previous similar message [ 4809.208760] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4809.252645] Lustre: Skipped 1 previous similar message [ 4814.311213] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4814.321218] Lustre: Skipped 1 previous similar message [ 4815.382780] Lustre: server umount lustre-OST0000 complete [ 4819.425200] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4819.428530] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4819.433517] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4819.459546] Lustre: Skipped 1 previous similar message [ 4824.545225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4824.547384] Lustre: Skipped 1 previous similar message [ 4825.727674] Lustre: server umount lustre-MDT0000 complete [ 4829.253676] LustreError: 237173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790963708 with bad export cookie 15139775511221880655 [ 4829.264641] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4829.266461] LustreError: 237173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4829.698800] Lustre: server umount lustre-MDT0001 complete [ 4844.759124] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 4852.994297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4853.256095] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4856.846184] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4858.337843] LustreError: 242080:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4858.368550] LustreError: 242080:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4863.464710] LustreError: 242081:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4863.811952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4864.082802] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4867.504347] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4869.745963] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4874.860543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4875.118590] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4880.091693] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4886.249974] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4886.424663] Lustre: lustre-OST0001: new disk, initializing [ 4886.427738] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4886.432399] Lustre: 244386:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4889.405623] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4889.420502] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4889.471941] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4891.492072] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 4901.516445] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4905.918881] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4930.260571] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 13:56:46 (1790963806) [ 4935.652431] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4935.665350] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4935.673962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4946.915605] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4946.918788] Lustre: Skipped 11 previous similar messages [ 4949.478670] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4949.854547] Lustre: server umount lustre-MDT0000 complete [ 4952.052994] LustreError: 242081:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4952.096447] LustreError: 242081:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4955.491542] LustreError: 242061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790963834 with bad export cookie 15139775511221881537 [ 4955.515642] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4955.516732] LustreError: 242061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4955.884493] Lustre: server umount lustre-MDT0001 complete [ 4970.147757] Lustre: server umount lustre-OST0000 complete [ 4983.709694] Lustre: server umount lustre-OST0001 complete [ 4992.257975] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [ 5004.252678] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 5040.088031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5040.331900] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5040.355064] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5040.409301] Lustre: lustre-MDT0000: new disk, initializing [ 5040.494570] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5040.499522] Lustre: Skipped 1 previous similar message [ 5040.510841] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5044.815352] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5053.789761] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5053.914680] Lustre: 248356:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5053.992161] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5053.995574] Lustre: Skipped 1 previous similar message [ 5054.044691] Lustre: lustre-MDT0001: new disk, initializing [ 5054.142476] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5054.148868] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5057.708414] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5067.351339] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5072.616186] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5079.800908] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5080.230804] Lustre: lustre-OST0000: new disk, initializing [ 5080.247583] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5080.266572] Lustre: 250321:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5080.452384] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5080.462946] Lustre: Skipped 1 previous similar message [ 5081.510551] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5081.548889] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5081.728962] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5087.757751] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5097.049741] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5101.494566] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5101.730069] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5106.679542] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5106.884854] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5112.300271] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5112.311732] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5112.319380] Lustre: Skipped 3 previous similar messages [ 5112.323827] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5112.328707] Lustre: Skipped 1 previous similar message [ 5115.301383] Lustre: server umount lustre-OST0000 complete [ 5120.992503] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5120.999034] LustreError: Skipped 1 previous similar message [ 5121.003628] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5121.011100] Lustre: Skipped 1 previous similar message [ 5125.641893] Lustre: server umount lustre-MDT0000 complete [ 5127.651725] LustreError: 248368:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5128.436973] LustreError: 248349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790964007 with bad export cookie 15139775511221883672 [ 5128.440468] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5128.453897] LustreError: 248349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5128.831481] Lustre: server umount lustre-MDT0001 complete [ 5138.195277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5138.957737] LustreError: 252540:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5139.002554] LustreError: 252540:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5145.035899] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5149.666100] LustreError: 252541:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5149.702876] LustreError: 252541:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5154.141964] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5154.423745] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5154.433125] Lustre: Skipped 1 previous similar message [ 5159.069785] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5166.818173] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5172.341694] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5179.212453] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5179.366718] LustreError: 254360:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5185.883851] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5194.580964] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5343.713536] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5343.714294] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5343.725038] LustreError: Skipped 1 previous similar message [ 5343.731902] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5343.757166] Lustre: Skipped 2 previous similar messages [ 5343.763622] Lustre: Skipped 3 previous similar messages [ 5347.567506] Lustre: server umount lustre-OST0000 complete [ 5364.193599] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5364.313132] Lustre: server umount lustre-MDT0000 complete [ 5366.857163] LustreError: 252520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790964245 with bad export cookie 15139775511221884554 [ 5366.859415] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5366.873026] LustreError: 252520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5367.067366] Lustre: server umount lustre-MDT0001 complete [ 5462.688779] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 5466.380582] Key type lgssc unregistered [ 5466.739626] LNet: 256162:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5466.754048] LNetError: 256162:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5466.779044] LNet: Removed LNI 192.168.202.121@tcp [ 5467.929546] Key type .llcrypt unregistered [ 5467.932181] Key type ._llcrypt unregistered [ 5489.243044] Key type ._llcrypt registered [ 5489.244697] Key type .llcrypt registered [ 5489.374250] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [ 5504.298569] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 5505.154991] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5505.305682] alg: No test for adler32 (adler32-zlib) [ 5506.331877] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 5506.541094] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 5508.184696] Key type lgssc registered [ 5509.180269] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5548.635737] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 5556.221312] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 5556.234933] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5557.376984] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5557.402373] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5557.447270] Lustre: lustre-MDT0000: new disk, initializing [ 5557.498725] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5557.510628] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5560.456289] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5569.469470] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5569.576893] Lustre: 260248:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5569.622875] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5569.626040] Lustre: Skipped 1 previous similar message [ 5569.679478] Lustre: lustre-MDT0001: new disk, initializing [ 5569.774674] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5569.806673] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5569.816910] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5574.053223] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5579.286657] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5584.842474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5585.032289] Lustre: lustre-OST0000: new disk, initializing [ 5585.034697] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5585.045153] Lustre: 261876:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5585.148721] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5585.372428] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5585.387951] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5585.433559] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 5591.283474] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5602.527960] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5602.657744] Lustre: lustre-OST0001: new disk, initializing [ 5602.664297] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5602.669735] Lustre: 262749:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5602.775187] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5608.744647] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5611.081563] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5611.095253] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5611.180413] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5619.223629] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5624.818695] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5636.576546] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5636.587104] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5636.598514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5637.092652] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5637.113528] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5637.129158] Lustre: Skipped 3 previous similar messages [ 5642.219609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5642.242771] Lustre: Skipped 3 previous similar messages [ 5647.346970] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5649.376448] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5649.810670] Lustre: server umount lustre-MDT0000 complete [ 5653.861209] LustreError: 261891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790964532 with bad export cookie 15110240073834655966 [ 5653.862328] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5653.868350] LustreError: 261891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5654.181925] Lustre: server umount lustre-MDT0001 complete [ 5668.803696] Lustre: server umount lustre-OST0000 complete [ 5683.354633] Lustre: server umount lustre-OST0001 complete [ 5698.562697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5709.395396] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5718.726698] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 5729.374769] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 5748.826187] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 5758.833096] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5758.847701] Lustre: lustre-MDT0000: reset Object Index mappings [ 5759.179449] LustreError: 267765:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5759.194549] LustreError: 267765:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 5759.235079] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5763.593575] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5764.589552] LustreError: 267766:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5769.697690] LustreError: 267765:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5772.120642] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5772.138749] Lustre: lustre-MDT0001: reset Object Index mappings [ 5772.650966] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5778.377558] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5781.444389] Lustre: 268905:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5788.623449] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5789.278770] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5797.539983] LustreError: 269262:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5800.378262] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5803.005436] LustreError: 269561:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5804.079832] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 5812.289784] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5817.884284] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 5818.149848] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5827.285281] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5831.133955] Lustre: 270782:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 5838.304890] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5838.317017] Lustre: Skipped 2 previous similar messages [ 5838.322383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5838.327563] Lustre: Skipped 5 previous similar messages [ 5843.439229] LustreError: 267765:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5843.471716] LustreError: 267765:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 5843.699407] Lustre: server umount lustre-MDT0000 complete [ 5849.239901] LustreError: 267747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790964728 with bad export cookie 15110240073834662749 [ 5849.243236] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5849.271662] LustreError: 267747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5850.240830] Lustre: server umount lustre-MDT0001 complete [ 5866.273763] Lustre: server umount lustre-OST0000 complete [ 5869.088797] Lustre: 257745:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790964732/real 1790964732] req@ffff9ae87ce94380 x1877962271096192/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790964748 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5869.137394] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5869.166722] Lustre: Skipped 1 previous similar message [ 5870.049797] Lustre: 257747:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790964733/real 1790964733] req@ffff9ae87731c380 x1877962271096448/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790964749 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5871.542941] Lustre: server umount lustre-OST0001 complete [ 5894.411889] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 14:12:51 (1790964771) [ 5914.561800] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [ 5926.793220] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 5968.354650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5968.822840] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5968.935911] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5968.980640] Lustre: lustre-MDT0000: new disk, initializing [ 5969.020594] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5969.022595] Lustre: Skipped 1 previous similar message [ 5969.031947] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5973.675590] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 5986.525945] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5986.715518] Lustre: 275037:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5986.777110] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5986.780909] Lustre: Skipped 1 previous similar message [ 5986.872375] Lustre: lustre-MDT0001: new disk, initializing [ 5987.067063] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5987.078569] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5993.209581] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6003.753866] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6010.258987] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6018.199929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6018.607690] Lustre: lustre-OST0000: new disk, initializing [ 6018.614644] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6018.619304] Lustre: 277008:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6019.778788] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6019.789213] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6019.934103] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6025.946100] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6036.795672] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6042.132428] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6042.377243] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6047.808857] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6048.067178] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6050.786654] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6050.788816] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6050.825039] Lustre: Skipped 1 previous similar message [ 6050.835021] Lustre: Skipped 1 previous similar message [ 6056.466539] Lustre: server umount lustre-OST0000 complete [ 6061.032597] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6061.059597] Lustre: Skipped 1 previous similar message [ 6071.273429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6071.279686] Lustre: Skipped 7 previous similar messages [ 6075.359156] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6075.757386] Lustre: server umount lustre-MDT0000 complete [ 6076.385377] LustreError: 275048:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6076.409887] LustreError: 275048:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 6081.638371] LustreError: 275031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790964960 with bad export cookie 15110240073834664849 [ 6081.643640] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6081.661130] LustreError: 275031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6082.616633] Lustre: server umount lustre-MDT0001 complete [ 6106.061545] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 6118.060483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6118.765936] LustreError: 279945:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6118.805566] LustreError: 279945:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6118.864153] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6118.867288] Lustre: Skipped 2 previous similar messages [ 6124.293806] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6134.614091] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6140.021878] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6143.615869] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6151.764468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6159.575269] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6171.269824] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6171.483872] Lustre: lustre-OST0001: new disk, initializing [ 6171.487248] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6171.492831] Lustre: 282253:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6173.096891] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6173.114439] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6173.258444] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6178.944431] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6189.814780] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6193.197703] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6200.383079] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 6200.525728] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 6214.121512] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6214.134921] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6214.148236] Lustre: Skipped 2 previous similar messages [ 6214.154586] Lustre: Skipped 1 previous similar message [ 6218.793174] Lustre: server umount lustre-MDT0000 complete [ 6219.242921] LustreError: 279941:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6219.264162] LustreError: 279941:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 6222.784054] LustreError: 279927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790965101 with bad export cookie 15110240073834665724 [ 6222.785448] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6222.789760] LustreError: 279927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6223.021266] Lustre: server umount lustre-MDT0001 complete [ 6236.758843] Lustre: server umount lustre-OST0000 complete [ 6240.543603] Lustre: 257747:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790965103/real 1790965103] req@ffff9ae7491fc700 x1877962271215616/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790965119 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6241.602730] Lustre: server umount lustre-OST0001 complete [ 6261.208180] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 6273.651901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6274.213101] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6274.223423] Lustre: Skipped 3 previous similar messages [ 6279.552889] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6289.361314] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6289.908476] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6295.738869] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6299.280148] Lustre: 286742:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6306.792102] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6314.903540] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6319.755844] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 6325.092828] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6330.885293] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 6332.953864] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6341.762253] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6346.375578] Lustre: 288616:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6353.730770] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 6353.814712] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 6368.993902] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 14:20:46 (1790965246) [ 6387.173042] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6387.187576] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6387.202888] Lustre: Skipped 3 previous similar messages [ 6391.621800] Lustre: server umount lustre-MDT0000 complete [ 6392.291602] LustreError: 285599:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6392.322180] LustreError: 285599:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [ 6396.098055] LustreError: 285578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790965275 with bad export cookie 15110240073834672619 [ 6396.104108] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6396.117797] LustreError: 285578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6402.555779] Lustre: server umount lustre-MDT0001 complete [ 6413.951673] Lustre: server umount lustre-OST0000 complete [ 6418.173780] Lustre: server umount lustre-OST0001 complete [ 6426.569937] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [ 6438.715200] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 6495.790378] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 6508.101412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6508.510046] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6508.567140] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6508.662858] Lustre: lustre-MDT0000: new disk, initializing [ 6508.880413] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6514.484381] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6527.279993] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6527.493633] Lustre: 293148:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6527.893970] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6534.146872] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6548.006047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6548.237055] Lustre: 294782:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6548.335433] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6548.342223] Lustre: Skipped 5 previous similar messages [ 6549.964588] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6556.712629] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6571.949197] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6572.247403] Lustre: 295652:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6574.226407] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6574.232120] Lustre: Skipped 1 previous similar message [ 6574.383907] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6581.916364] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6594.888520] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6599.245831] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6599.249591] Lustre: Skipped 1 previous similar message [ 6616.497397] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 14:24:54 (1790965494) [ 6620.129033] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6620.134807] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6620.166270] Lustre: Skipped 3 previous similar messages [ 6620.176623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6620.179029] Lustre: Skipped 8 previous similar messages [ 6621.717494] Lustre: server umount lustre-MDT0000 complete [ 6625.933973] LustreError: 293141:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790965505 with bad export cookie 15110240073834680088 [ 6625.941527] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6625.945952] LustreError: 293141:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6626.355424] Lustre: server umount lustre-MDT0001 complete [ 6640.183116] Lustre: server umount lustre-OST0000 complete [ 6654.580402] Lustre: server umount lustre-OST0001 complete [ 6665.840376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6666.283703] LustreError: 297901:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6666.307943] LustreError: 297901:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [ 6672.239975] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6686.182494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6693.713213] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6703.049971] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6709.760838] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6717.632435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6725.019899] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6733.846514] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6756.725860] loop1: detected capacity change from 0 to 204800000 [ 6757.157430] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6757.185221] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6757.209261] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6763.431619] loop1: detected capacity change from 0 to 204800000 [ 6770.373965] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6770.607805] LustreError: 297890:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs13090: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 6770.626847] LustreError: 297890:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs13090 [ 6770.636321] LustreError: 297890:0:(mgs_handler.c:517:mgs_target_reg()) fs13090-MDT0001: Error checking target index: rc = -13 [ 6770.655270] LustreError: fs13090-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 6770.667468] LustreError: 301626:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [ 6770.679696] LustreError: 301626:0:(tgt_mount.c:2128:server_put_super()) no obd fs13090-MDT0001 [ 6770.687741] LustreError: 301626:0:(tgt_mount.c:116:server_deregister_mount()) fs13090-MDT0001 not registered [ 6770.784291] Lustre: server umount fs13090-MDT0001 complete [ 6770.793117] LustreError: 301626:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 6782.173583] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6782.242665] Lustre: Setting parameter fs13090-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs13090-MDT0001 [ 6782.276911] Lustre: srv-fs13090-MDT0001: No data found on store. Initialize space. [ 6782.281138] Lustre: Skipped 4 previous similar messages [ 6782.344571] Lustre: fs13090-MDT0001: new disk, initializing [ 6782.346691] Lustre: Skipped 3 previous similar messages [ 6782.476675] Lustre: srv-fs13090-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 6782.483671] Lustre: 302093:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs13090-MDT0001-mdtlov: can't create new llog: rc = -115 [ 6782.494726] LustreError: 302093:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs13090-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 6788.723907] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6797.268980] Lustre: server umount fs13090-MDT0001 complete [ 6811.569965] loop1: detected capacity change from 0 to 204800000 [ 6813.681553] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [ 6818.074944] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6818.368351] Lustre: 303662:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs13090-MDT0001-mdtlov: can't create new llog: rc = -115 [ 6818.385325] LustreError: 303662:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs13090-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 6822.808333] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6828.486160] Lustre: server umount fs13090-MDT0001 complete [ 6844.393951] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6844.423049] Lustre: Skipped 3 previous similar messages [ 6848.420696] Lustre: server umount lustre-MDT0000 complete [ 6859.318276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6859.475807] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6864.932027] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6865.416789] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6874.376565] loop1: detected capacity change from 0 to 204800000 [ 6881.272097] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6881.984922] Lustre: 306109:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs13090-MDT0001-mdtlov: can't create new llog: rc = -115 [ 6881.996440] LustreError: 306109:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs13090-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 6889.142229] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 6895.212217] Lustre: server umount fs13090-MDT0001 complete [ 6921.188149] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6921.193493] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6921.200372] LustreError: Skipped 2 previous similar messages [ 6921.218983] Lustre: Skipped 9 previous similar messages [ 6927.035284] Lustre: server umount lustre-OST0000 complete [ 6936.544561] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6938.082814] Lustre: server umount lustre-MDT0000 complete [ 6941.785236] LustreError: 297882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790965820 with bad export cookie 15110240073834682769 [ 6941.814546] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6942.206656] Lustre: server umount lustre-MDT0001 complete [ 6954.064882] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 6956.903953] Key type lgssc unregistered [ 6957.185843] LNet: 308456:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6957.194069] LNetError: 308456:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6957.209373] LNet: Removed LNI 192.168.202.121@tcp [ 6958.085541] Key type .llcrypt unregistered [ 6958.090407] Key type ._llcrypt unregistered [ 6979.558150] Key type ._llcrypt registered [ 6979.559716] Key type .llcrypt registered [ 6979.637842] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 6980.750977] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6980.823621] alg: No test for adler32 (adler32-zlib) [ 6982.081502] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 6982.462372] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 6984.135234] Key type lgssc registered [ 6985.436562] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6996.292932] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 6996.329535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6998.139492] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7005.041351] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 7008.748792] LustreError: 309742:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7008.770679] LustreError: 309742:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7013.865760] LustreError: 309741:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7016.694952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7017.056945] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7022.915995] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 7030.673506] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7036.300681] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7043.590699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7043.986691] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7050.212939] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 7058.825492] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7062.446833] LustreError: 311572:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7067.624082] LustreError: 311574:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7073.770534] LustreError: 311574:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7073.803834] LustreError: 311574:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 7084.026402] LustreError: 311572:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7084.053314] LustreError: 311572:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 7092.079370] loop1: detected capacity change from 0 to 204800000 [ 7092.415220] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7092.445228] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7092.461699] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7098.986464] loop1: detected capacity change from 0 to 204800000 [ 7104.493346] LustreError: 311573:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7104.516039] LustreError: 311573:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [ 7105.462920] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7105.717664] Lustre: fs9058-OST0000: new disk, initializing [ 7105.721604] Lustre: srv-fs9058-OST0000: No data found on store. Initialize space. [ 7105.728362] Lustre: 313502:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7105.826931] Lustre: fs9058-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7112.955840] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 7123.279386] Lustre: server umount fs9058-OST0000 complete [ 7138.660428] loop1: detected capacity change from 0 to 204800000 [ 7140.326930] LustreError: 312243:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7140.329282] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [ 7140.340437] LustreError: 312243:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 20 previous similar messages [ 7145.457379] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [ 7146.320550] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7146.619292] Lustre: fs9058-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7156.599228] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 7167.868288] loop2: detected capacity change from 0 to 204800000 [ 7168.256231] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7168.284017] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7168.302588] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7174.874212] loop2: detected capacity change from 0 to 204800000 [ 7181.545158] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7181.617495] LustreError: fs9058-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [ 7181.626180] LustreError: fs9058-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 7181.637731] LustreError: 316542:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [ 7181.642548] LustreError: 316542:0:(tgt_mount.c:2128:server_put_super()) no obd fs9058-OST0001 [ 7181.649253] LustreError: 316542:0:(tgt_mount.c:116:server_deregister_mount()) fs9058-OST0001 not registered [ 7181.686658] Lustre: server umount fs9058-OST0001 complete [ 7181.689320] LustreError: 316542:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 7192.382963] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7192.497327] Lustre: fs9058-OST0001: new disk, initializing [ 7192.501859] Lustre: srv-fs9058-OST0001: No data found on store. Initialize space. [ 7192.508247] Lustre: 317093:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7192.566954] Lustre: fs9058-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7199.562718] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 7206.881738] LustreError: 311574:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7206.918221] LustreError: 311574:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 38 previous similar messages [ 7209.622875] Lustre: server umount fs9058-OST0000 complete [ 7221.100845] Lustre: server umount fs9058-OST0001 complete [ 7236.075613] loop1: detected capacity change from 0 to 204800000 [ 7246.257923] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7246.694876] Lustre: fs9058-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7255.189723] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 7263.130423] loop2: detected capacity change from 0 to 204800000 [ 7269.475929] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7276.474963] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 7287.031717] Lustre: server umount fs9058-OST0000 complete [ 7293.938647] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [ 7297.970650] Lustre: server umount fs9058-OST0001 complete [ 7299.056761] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [ 7313.378926] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7313.383232] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7313.407439] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7314.408379] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7314.415303] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7318.882574] Lustre: server umount lustre-OST0000 complete [ 7324.128341] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7324.135313] LustreError: Skipped 1 previous similar message [ 7324.138213] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7324.153237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7324.156023] Lustre: Skipped 1 previous similar message [ 7328.827548] Lustre: server umount lustre-MDT0000 complete [ 7332.627447] LustreError: 309722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790966211 with bad export cookie 1737398960523236177 [ 7332.631207] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7332.639117] LustreError: 309722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7333.014566] Lustre: server umount lustre-MDT0001 complete [ 7344.558966] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 7347.554396] Key type lgssc unregistered [ 7347.831683] LNet: 323018:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7347.837330] LNetError: 323018:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7348.903205] LNet: Removed LNI 192.168.202.121@tcp [ 7349.996173] Key type .llcrypt unregistered [ 7350.000198] Key type ._llcrypt unregistered [ 7386.588752] Key type ._llcrypt registered [ 7386.595899] Key type .llcrypt registered [ 7386.728769] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [ 7405.191962] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 7406.844208] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7406.980774] alg: No test for adler32 (adler32-zlib) [ 7408.070784] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 7408.293168] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 7409.912066] Key type lgssc registered [ 7410.877566] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7454.824967] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 7454.855103] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7456.235925] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7456.250764] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7456.412382] Lustre: lustre-MDT0000: new disk, initializing [ 7456.510739] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7456.530215] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7461.449494] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 7473.140833] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7473.343416] Lustre: 326896:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7473.432665] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7473.439619] Lustre: Skipped 1 previous similar message [ 7473.545345] Lustre: lustre-MDT0001: new disk, initializing [ 7473.754608] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7473.837299] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7473.859768] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7478.161093] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 7487.683666] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7493.963891] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7502.002563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7502.458831] Lustre: lustre-OST0000: new disk, initializing [ 7502.470454] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7502.482735] Lustre: 328867:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7502.627922] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7508.055217] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7508.069872] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7508.168878] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7510.050359] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 7520.911594] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7523.041144] Lustre: Mounted lustre-client - version 2.17.59_2_gbfaf0e7 [ 7524.188419] Lustre: Unmounted lustre-client [ 7533.553885] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7533.574140] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7533.591818] Lustre: Skipped 1 previous similar message [ 7538.667250] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7538.673113] Lustre: Skipped 1 previous similar message [ 7539.610994] Lustre: server umount lustre-OST0000 complete [ 7543.778600] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7543.797137] Lustre: Skipped 1 previous similar message [ 7543.806057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7543.816273] Lustre: Skipped 1 previous similar message [ 7548.898536] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7548.913239] Lustre: Skipped 1 previous similar message [ 7549.889064] Lustre: server umount lustre-MDT0000 complete [ 7553.845462] LustreError: 326890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790966432 with bad export cookie 17965131046750833855 [ 7553.861324] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7553.862136] LustreError: 326890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7554.194722] Lustre: server umount lustre-MDT0001 complete [ 7566.503658] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 7569.068524] Key type lgssc unregistered [ 7577.508652] LNet: 331024:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7577.519606] LNetError: 331024:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7652.345120] LNet: Removed LNI 192.168.202.121@tcp [ 7653.478318] Key type .llcrypt unregistered [ 7653.486911] Key type ._llcrypt unregistered [ 7675.083482] Key type ._llcrypt registered [ 7675.085048] Key type .llcrypt registered [ 7675.309396] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [ 7695.506600] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 7696.490850] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7696.536518] alg: No test for adler32 (adler32-zlib) [ 7697.947867] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 7698.251393] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 7699.903174] Key type lgssc registered [ 7700.856065] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7733.992619] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 14:43:31 (1790966611) [ 7740.692574] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 7740.717417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7741.940053] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7741.983742] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7742.057915] Lustre: lustre-MDT0000: new disk, initializing [ 7742.162884] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7742.189839] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7747.429495] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 7757.740912] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7757.825642] Lustre: 334263:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7757.877097] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7757.880828] Lustre: Skipped 1 previous similar message [ 7757.924645] Lustre: lustre-MDT0001: new disk, initializing [ 7757.998440] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7758.033038] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7758.041834] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7761.367876] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 7773.462481] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7781.555483] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7790.957975] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7791.263240] Lustre: lustre-OST0000: new disk, initializing [ 7791.272819] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7791.282775] Lustre: 336234:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7791.377550] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7796.822293] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7796.835963] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7796.990670] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7797.928629] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 7807.039982] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7822.823032] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7822.824419] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7822.829585] LustreError: Skipped 1 previous similar message [ 7822.866218] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7822.870912] Lustre: Skipped 1 previous similar message [ 7827.937612] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7827.943801] Lustre: Skipped 1 previous similar message [ 7828.352215] Lustre: server umount lustre-OST0000 complete [ 7833.060050] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7833.071235] Lustre: Skipped 1 previous similar message [ 7833.080017] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7833.089938] Lustre: Skipped 1 previous similar message [ 7838.177726] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7838.196291] Lustre: Skipped 1 previous similar message [ 7843.302884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7843.308901] Lustre: Skipped 1 previous similar message [ 7846.367427] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7846.526655] Lustre: server umount lustre-MDT0000 complete [ 7850.450993] LustreError: 334256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790966729 with bad export cookie 125742601485250934 [ 7850.457506] LustreError: 334256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7850.458865] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7850.802806] Lustre: server umount lustre-MDT0001 complete [ 7862.338077] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 7865.230845] Key type lgssc unregistered [ 7865.493054] LNet: 338142:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7865.505453] LNetError: 338142:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7865.530203] LNet: Removed LNI 192.168.202.121@tcp [ 7866.515306] Key type .llcrypt unregistered [ 7866.518332] Key type ._llcrypt unregistered [ 7887.247526] Key type ._llcrypt registered [ 7887.251717] Key type .llcrypt registered [ 7887.399216] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 7888.817548] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7888.860855] alg: No test for adler32 (adler32-zlib) [ 7890.181794] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 7890.367944] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 7892.007255] Key type lgssc registered [ 7893.733148] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7903.014850] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 7903.069979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7904.554426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7910.948360] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 7913.963937] LustreError: 339430:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7913.983558] LustreError: 339430:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7919.081682] LustreError: 339431:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7920.312172] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7920.834526] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7926.070204] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 7933.910249] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7940.013232] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7947.265336] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7947.553987] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7954.315862] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 7962.092084] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7975.371231] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7976.774856] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7980.521894] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7980.529729] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7980.538326] Lustre: Skipped 2 previous similar messages [ 7980.541689] Lustre: Skipped 1 previous similar message [ 7984.413481] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.21@tcp (stopping) [ 7984.422492] Lustre: Skipped 1 previous similar message [ 7985.637091] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7985.646044] Lustre: Skipped 2 previous similar messages [ 7986.856274] Lustre: server umount lustre-MDT0000 complete [ 7989.511669] LustreError: 339425:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7990.904537] LustreError: 339455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790966870 with bad export cookie 7782727501152141269 [ 7990.905564] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7990.925872] LustreError: 339455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7994.623372] LustreError: 339425:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7994.650331] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.21@tcp (stopping) [ 7994.651119] LustreError: 339425:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 7995.877790] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7997.397990] Lustre: server umount lustre-MDT0001 complete [ 8001.846247] Lustre: server umount lustre-OST0000 complete [ 8011.006267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8015.538652] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 8017.941425] Lustre: 343609:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.121@tcp [ 8019.802226] Lustre: server umount MGS complete [ 8028.009865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8028.488288] LustreError: 344181:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8028.511022] LustreError: 344181:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8028.606478] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8033.652501] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 8037.922669] LustreError: 344182:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8037.946496] LustreError: 344182:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8041.664530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8041.958549] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8046.953453] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 8057.119849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8057.780514] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8065.992881] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 8075.497258] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8077.649988] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [ 8085.352813] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8087.785660] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8093.665628] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8093.668547] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8093.678247] Lustre: Skipped 1 previous similar message [ 8093.690341] Lustre: Skipped 2 previous similar messages [ 8099.887670] Lustre: server umount lustre-OST0000 complete [ 8108.515639] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8108.526349] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8108.540259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8108.542799] Lustre: Skipped 2 previous similar messages [ 8110.959427] Lustre: server umount lustre-MDT0000 complete [ 8114.146175] LustreError: 344177:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8114.176522] LustreError: 344177:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8115.030435] LustreError: 344161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790966994 with bad export cookie 7782727501152142109 [ 8115.036095] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8115.044780] LustreError: 344161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8115.416754] Lustre: server umount lustre-MDT0001 complete [ 8128.244769] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 8131.698728] Key type lgssc unregistered [ 8132.023544] LNet: 347619:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8132.041497] LNetError: 347619:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8132.075255] LNet: Removed LNI 192.168.202.121@tcp [ 8132.869981] Key type .llcrypt unregistered [ 8132.871326] Key type ._llcrypt unregistered [ 8151.878155] Key type ._llcrypt registered [ 8151.881996] Key type .llcrypt registered [ 8152.114603] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [ 8173.257491] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 8174.222480] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8174.279188] alg: No test for adler32 (adler32-zlib) [ 8175.340514] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 8175.712304] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 8177.506863] Key type lgssc registered [ 8179.108979] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8235.134881] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 8235.199896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8236.697422] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8236.751540] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8236.931821] Lustre: lustre-MDT0000: new disk, initializing [ 8237.125479] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8237.173327] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8242.656595] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 8254.140696] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8254.276804] Lustre: 350799:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8254.333710] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8254.336751] Lustre: Skipped 1 previous similar message [ 8254.410554] Lustre: lustre-MDT0001: new disk, initializing [ 8254.500484] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8254.546339] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8254.568656] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8259.685499] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 8269.017872] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8274.194481] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8280.117091] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8280.327870] Lustre: lustre-OST0000: new disk, initializing [ 8280.330332] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8280.334088] Lustre: 352768:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8280.398632] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8286.642577] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 8289.877399] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8289.884588] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8289.993992] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8296.299919] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8301.102613] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8301.402876] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8304.994382] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8305.215880] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8310.754231] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8310.764544] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8310.780732] LustreError: Skipped 1 previous similar message [ 8310.784524] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8310.813829] Lustre: Skipped 1 previous similar message [ 8313.812837] Lustre: server umount lustre-OST0000 complete [ 8320.999785] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8321.005106] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8321.025350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8321.025360] Lustre: Skipped 1 previous similar message [ 8323.691362] Lustre: server umount lustre-MDT0000 complete [ 8329.066381] LustreError: 350794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790967208 with bad export cookie 6491133320694325231 [ 8329.073801] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8329.085836] LustreError: 350794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8329.658821] Lustre: server umount lustre-MDT0001 complete [ 8344.187470] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 14:53:40 (1790967220) [ 8363.769259] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [ 8374.358925] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 8410.625785] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8410.858523] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8410.895422] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8410.953515] Lustre: lustre-MDT0000: new disk, initializing [ 8411.007371] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8411.019262] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8415.902759] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 8426.530470] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8426.707871] Lustre: 357532:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8426.826699] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8426.839822] Lustre: Skipped 1 previous similar message [ 8426.991806] Lustre: lustre-MDT0001: new disk, initializing [ 8427.123847] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8427.175935] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8427.185521] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8432.734347] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 8442.780062] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8448.598127] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8454.613833] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8454.986300] Lustre: lustre-OST0000: new disk, initializing [ 8454.992402] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8454.997634] Lustre: 359498:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8455.145338] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8461.745654] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 8462.887875] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8462.902962] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8463.025485] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8469.783972] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8504.292230] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8504.293767] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8504.308218] LustreError: Skipped 1 previous similar message [ 8504.326580] Lustre: Skipped 1 previous similar message [ 8504.332250] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8504.345294] Lustre: Skipped 1 previous similar message [ 8505.709767] Lustre: server umount lustre-OST0000 complete [ 8509.410753] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8509.412968] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8509.440254] Lustre: Skipped 1 previous similar message [ 8509.462895] Lustre: Skipped 1 previous similar message [ 8514.530306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8514.538195] Lustre: Skipped 1 previous similar message [ 8515.846943] Lustre: server umount lustre-MDT0000 complete [ 8519.029212] LustreError: 359508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790967398 with bad export cookie 6491133320694326113 [ 8519.035080] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8519.036306] LustreError: 359508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8519.275737] Lustre: server umount lustre-MDT0001 complete [ 8530.270567] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 8533.135486] Key type lgssc unregistered [ 8533.422618] LNet: 361667:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8533.431056] LNetError: 361667:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8533.446185] LNet: Removed LNI 192.168.202.121@tcp [ 8533.925491] Key type .llcrypt unregistered [ 8533.929453] Key type ._llcrypt unregistered [ 8562.741073] Key type ._llcrypt registered [ 8562.742701] Key type .llcrypt registered [ 8562.850715] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 8563.565976] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8563.791842] alg: No test for adler32 (adler32-zlib) [ 8564.888884] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 8565.067593] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 8566.687213] Key type lgssc registered [ 8568.432869] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8577.296079] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 8577.328418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8578.809523] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8582.447223] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 8588.264171] LustreError: 362959:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8588.317860] LustreError: 362959:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8592.295151] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8592.855755] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8598.167617] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 8606.578346] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8616.675188] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8626.354533] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8626.959680] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8634.900957] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 8643.273072] Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8705.000576] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8705.002176] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8705.008885] Lustre: Skipped 1 previous similar message [ 8705.015831] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8710.116304] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8710.127424] Lustre: Skipped 1 previous similar message [ 8715.243179] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8715.253052] Lustre: Skipped 1 previous similar message [ 8719.327760] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8719.588372] Lustre: server umount lustre-OST0000 complete [ 8725.477907] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8725.486761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8725.505270] Lustre: Skipped 1 previous similar message [ 8725.525085] Lustre: Skipped 1 previous similar message [ 8729.732983] Lustre: server umount lustre-MDT0000 complete [ 8730.595875] LustreError: 362960:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8730.606176] LustreError: 362960:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8733.432852] LustreError: 362941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790967612 with bad export cookie 6898181490448968477 [ 8733.447706] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8733.894373] Lustre: server umount lustre-MDT0001 complete [ 8746.845956] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 8749.337699] Key type lgssc unregistered [ 8749.695872] LNet: 368569:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8749.702237] LNetError: 368569:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8749.719059] LNet: Removed LNI 192.168.202.121@tcp [ 8750.746420] Key type .llcrypt unregistered [ 8750.747952] Key type ._llcrypt unregistered [ 8856.595261] Key type ._llcrypt registered [ 8856.597696] Key type .llcrypt registered [ 8856.694639] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 8856.846443] Key type .llcrypt unregistered [ 8856.847639] Key type ._llcrypt unregistered [ 8947.418203] Key type ._llcrypt registered [ 8947.421590] Key type .llcrypt registered [ 8947.529770] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 8947.694336] Key type .llcrypt unregistered [ 8947.696159] Key type ._llcrypt unregistered [ 8978.849577] Key type ._llcrypt registered [ 8978.852328] Key type .llcrypt registered [ 8978.948860] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 8979.221392] Key type .llcrypt unregistered [ 8979.230341] Key type ._llcrypt unregistered [ 9012.317792] Key type ._llcrypt registered [ 9012.320129] Key type .llcrypt registered [ 9012.432337] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 9012.672317] Key type .llcrypt unregistered [ 9012.674063] Key type ._llcrypt unregistered [ 9086.892498] Key type ._llcrypt registered [ 9086.893706] Key type .llcrypt registered [ 9086.995683] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 9087.226375] Key type .llcrypt unregistered [ 9087.228585] Key type ._llcrypt unregistered [ 9129.542580] Key type ._llcrypt registered [ 9129.546131] Key type .llcrypt registered [ 9129.674328] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 9130.212765] Key type .llcrypt unregistered [ 9130.215282] Key type ._llcrypt unregistered [ 9160.825766] Key type ._llcrypt registered [ 9160.827364] Key type .llcrypt registered [ 9160.913408] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 9161.154282] Key type .llcrypt unregistered [ 9161.155447] Key type ._llcrypt unregistered [ 9197.658813] Key type ._llcrypt registered [ 9197.660105] Key type .llcrypt registered [ 9197.740895] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 9198.026680] Key type .llcrypt unregistered [ 9198.032444] Key type ._llcrypt unregistered [ 9232.310678] Key type ._llcrypt registered [ 9232.314698] Key type .llcrypt registered [ 9232.409831] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 9232.868089] Key type .llcrypt unregistered [ 9232.870674] Key type ._llcrypt unregistered [ 9267.947366] Key type ._llcrypt registered [ 9267.955379] Key type .llcrypt registered [ 9268.051318] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 9268.295709] Key type .llcrypt unregistered [ 9268.297667] Key type ._llcrypt unregistered [ 9325.581212] Key type ._llcrypt registered [ 9325.585867] Key type .llcrypt registered [ 9325.698600] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 9326.115304] Key type .llcrypt unregistered [ 9326.125848] Key type ._llcrypt unregistered [ 9355.070080] Key type ._llcrypt registered [ 9355.075295] Key type .llcrypt registered [ 9355.278974] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 9355.514306] Key type .llcrypt unregistered [ 9355.517041] Key type ._llcrypt unregistered [ 9390.537646] Key type ._llcrypt registered [ 9390.539942] Key type .llcrypt registered [ 9390.665825] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 9391.038360] Key type .llcrypt unregistered [ 9391.039722] Key type ._llcrypt unregistered [ 9448.189057] Key type ._llcrypt registered [ 9448.190474] Key type .llcrypt registered [ 9448.343575] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [ 9466.313918] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 9467.830294] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9467.860293] alg: No test for adler32 (adler32-zlib) [ 9468.956896] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 9469.275417] LNet: Added LNI 192.168.202.121@tcp [8/256/0/180] [ 9470.993230] Key type lgssc registered [ 9472.333897] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9503.793571] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 9503.833654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9505.195468] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [ 9505.228589] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [ 9505.305635] Lustre: tst250fs-MDT0000: new disk, initializing [ 9505.368195] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9505.380566] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9510.248604] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 9518.849741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9518.974620] Lustre: 388933:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9519.031098] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [ 9519.035399] Lustre: Skipped 1 previous similar message [ 9519.114647] Lustre: tst250fs-MDT0001: new disk, initializing [ 9519.197454] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9519.250092] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9519.274729] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9523.613799] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 9528.620985] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9534.145865] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9534.579997] Lustre: tst250fs-OST0000: new disk, initializing [ 9534.584277] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [ 9534.591875] Lustre: 390563:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9534.735479] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9538.579254] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9538.604427] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9538.808064] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9543.079617] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 9557.048120] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9557.260319] Lustre: tst250fs-OST0001: new disk, initializing [ 9557.266760] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [ 9557.276461] Lustre: 391437:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9557.462378] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9563.761656] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug -1 all [ 9566.832162] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9566.856103] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9566.982081] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9569.252616] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [ 9571.899527] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [ 9571.904555] Lustre: Skipped 2 previous similar messages [ 9590.752754] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9590.758675] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9590.776892] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 9592.806623] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9592.807918] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 9592.840740] Lustre: Skipped 3 previous similar messages [ 9594.305522] Lustre: server umount tst250fs-MDT0000 complete [ 9597.922549] LustreError: 388925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790968477 with bad export cookie 10206214549359317142 [ 9597.946541] LustreError: 388925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9597.951608] LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9597.962480] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9597.966405] Lustre: tst250fs-MDT0001: Not available for connect from 0@lo (stopping) [ 9597.982487] Lustre: Skipped 3 previous similar messages [ 9597.997825] Lustre: Skipped 1 previous similar message [ 9598.313620] Lustre: server umount tst250fs-MDT0001 complete [ 9607.957688] Lustre: server umount tst250fs-OST0000 complete [ 9612.099206] Lustre: server umount tst250fs-OST0001 complete [ 9629.830367] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [ 9637.764785] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 9668.300045] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 15:15:46 (1790968546) [ 9669.872560] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [ 9685.083637] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_hostid [ 9696.239280] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing load_modules_local [ 9722.498271] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 9543 sec ========== 15:16:40 (1790968600) [ 9724.518735] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 15:16:42 (1790968602) === [ 9728.294657] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 15:16:45 (1790968605) === [ 9767.221368] Lustre: DEBUG MARKER: oleg221-server.virtnet: executing unload_modules_local [ 9770.390351] Key type lgssc unregistered [ 9770.736734] LNet: 400575:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9770.748637] LNetError: 400575:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9770.763823] LNet: Removed LNI 192.168.202.121@tcp [ 9771.407717] Key type .llcrypt unregistered [ 9771.413881] Key type ._llcrypt unregistered