[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 759180056 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2835848K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003058] x2apic enabled [ 0.004011] Switched APIC routing to physical x2apic. [ 0.005018] kvm-guest: setup PV IPIs [ 0.008751] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009025] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010015] pid_max: default: 32768 minimum: 301 [ 0.011132] LSM: Security Framework initializing [ 0.012066] Yama: becoming mindful. [ 0.013063] SELinux: Initializing. [ 0.014097] *** VALIDATE selinux *** [ 0.022829] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027132] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028156] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029137] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030122] *** VALIDATE tmpfs *** [ 0.032260] *** VALIDATE proc *** [ 0.034003] *** VALIDATE cgroup *** [ 0.035051] *** VALIDATE cgroup2 *** [ 0.037032] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038171] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040036] Spectre V2 : User space: Vulnerable [ 0.041013] Speculative Store Bypass: Vulnerable [ 0.044186] debug: unmapping init [mem 0xffffffff8cc59000-0xffffffff8cc60fff] [ 0.046296] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047800] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048031] ... version: 2 [ 0.049013] ... bit width: 48 [ 0.050015] ... generic registers: 4 [ 0.051017] ... value mask: 0000ffffffffffff [ 0.052023] ... max period: 00007fffffffffff [ 0.053029] ... fixed-purpose events: 3 [ 0.054020] ... event mask: 000000070000000f [ 0.055364] rcu: Hierarchical SRCU implementation. [ 0.057739] smp: Bringing up secondary CPUs ... [ 0.058721] x86: Booting SMP configuration: [ 0.059036] .... node #0, CPUs: #1 #2 #3 [ 0.067192] smp: Brought up 1 node, 4 CPUs [ 0.069015] smpboot: Max logical packages: 1 [ 0.070022] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.109074] node 0 deferred pages initialised in 38ms [ 0.121577] devtmpfs: initialized [ 0.122000] x86/mm: Memory block size: 128MB [ 0.126468] gcov: version magic: 0x41383552 [ 0.130330] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.134190] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.138425] pinctrl core: initialized pinctrl subsystem [ 0.141204] [ 0.142009] ************************************************************* [ 0.145013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.148015] ** ** [ 0.151019] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.154016] ** ** [ 0.155018] ** This means that this kernel is built to expose internal ** [ 0.160017] ** IOMMU data structures, which may compromise security on ** [ 0.163015] ** your system. ** [ 0.164015] ** ** [ 0.169060] ** If you see this message and you are not debugging the ** [ 0.174014] ** kernel, report this immediately to your vendor! ** [ 0.176015] ** ** [ 0.179015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.182017] ************************************************************* [ 0.187898] NET: Registered protocol family 16 [ 0.189543] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.194075] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.198087] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.214147] cpuidle: using governor menu [ 0.219140] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.222723] PCI: Using configuration type 1 for base access [ 0.226248] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.233423] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.235019] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.240068] cryptd: max_cpu_qlen set to 1000 [ 0.243276] ACPI: Added _OSI(Module Device) [ 0.245012] ACPI: Added _OSI(Processor Device) [ 0.246011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.248011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.253939] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.264508] ACPI: Interpreter enabled [ 0.266078] ACPI: PM: (supports S0 S3 S4 S5) [ 0.269016] ACPI: Using IOAPIC for interrupt routing [ 0.275137] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.282549] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.301485] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.305064] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.309019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.314236] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.323000] acpiphp: Slot [2] registered [ 0.325209] acpiphp: Slot [5] registered [ 0.327164] acpiphp: Slot [6] registered [ 0.329182] acpiphp: Slot [7] registered [ 0.331154] acpiphp: Slot [8] registered [ 0.333327] acpiphp: Slot [9] registered [ 0.336169] acpiphp: Slot [10] registered [ 0.337254] acpiphp: Slot [3] registered [ 0.339473] acpiphp: Slot [4] registered [ 0.343158] acpiphp: Slot [11] registered [ 0.346169] acpiphp: Slot [12] registered [ 0.348174] acpiphp: Slot [13] registered [ 0.351386] acpiphp: Slot [14] registered [ 0.354328] acpiphp: Slot [15] registered [ 0.357213] acpiphp: Slot [16] registered [ 0.359158] acpiphp: Slot [17] registered [ 0.361342] acpiphp: Slot [18] registered [ 0.364159] acpiphp: Slot [19] registered [ 0.366141] acpiphp: Slot [20] registered [ 0.368134] acpiphp: Slot [21] registered [ 0.371122] acpiphp: Slot [22] registered [ 0.373234] acpiphp: Slot [23] registered [ 0.375115] acpiphp: Slot [24] registered [ 0.377145] acpiphp: Slot [25] registered [ 0.379181] acpiphp: Slot [26] registered [ 0.381238] acpiphp: Slot [27] registered [ 0.384530] acpiphp: Slot [28] registered [ 0.388137] acpiphp: Slot [29] registered [ 0.391198] acpiphp: Slot [30] registered [ 0.394146] acpiphp: Slot [31] registered [ 0.396088] PCI host bridge to bus 0000:00 [ 0.400025] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.405025] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.410072] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.416039] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.422030] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.428027] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.432820] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.439093] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.444254] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.460017] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.468065] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.472639] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.476022] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.480020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.486236] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.491097] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.496239] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.502529] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.511016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.535017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.550018] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.561000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.573017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.582017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.600016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.614356] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.627017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.635015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.659081] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.707439] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.720015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.758000] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.901019] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.916271] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.947025] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.950000] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.950000] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.958000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 1.015016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 1.033019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 1.056018] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 1.075325] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 1.085075] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 1.097016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 1.119016] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 1.239000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 1.239000] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 1.239000] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 1.239000] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 1.239000] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 1.239129] iommu: Default domain type: Passthrough [ 1.240733] SCSI subsystem initialized [ 1.241195] ACPI: bus type USB registered [ 1.242118] usbcore: registered new interface driver usbfs [ 1.243114] usbcore: registered new interface driver hub [ 1.244128] usbcore: registered new device driver usb [ 1.245000] pps_core: LinuxPPS API ver. 1 registered [ 1.245000] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 1.251094] PTP clock support registered [ 1.257360] EDAC MC: Ver: 3.0.0 [ 1.316422] PCI: Using ACPI for IRQ routing [ 1.317908] NetLabel: Initializing [ 1.319017] NetLabel: domain hash size = 128 [ 1.322015] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.325098] NetLabel: unlabeled traffic allowed by default [ 1.397200] vgaarb: loaded [ 1.402044] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.434017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.437885] clocksource: Switched to clocksource kvm-clock [ 1.639491] VFS: Disk quotas dquot_6.6.0 [ 1.643340] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.646107] *** VALIDATE ramfs *** [ 1.650515] *** VALIDATE hugetlbfs *** [ 1.652296] pnp: PnP ACPI init [ 1.654934] pnp: PnP ACPI: found 6 devices [ 1.671724] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.675027] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.677179] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.679517] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.682359] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.685059] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.688291] NET: Registered protocol family 2 [ 1.690969] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.696558] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.700637] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.730666] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.743258] TCP: Hash tables configured (established 65536 bind 65536) [ 1.753849] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.762599] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.770236] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.776789] NET: Registered protocol family 1 [ 1.785264] RPC: Registered named UNIX socket transport module. [ 1.790880] RPC: Registered udp transport module. [ 1.797501] RPC: Registered tcp transport module. [ 1.802904] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.810896] NET: Registered protocol family 44 [ 1.816220] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.818501] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.823955] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.830360] PCI: CLS 0 bytes, default 64 [ 1.834798] Unpacking initramfs... [ 5.661819] debug: unmapping init [mem 0xffff9e9dfcc54000-0xffff9e9dfffbffff] [ 5.666385] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 5.669069] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 5.673605] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 6.234353] Initialise system trusted keyrings [ 6.236103] Key type blacklist registered [ 6.237920] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 6.247221] zbud: loaded [ 6.250146] *** VALIDATE nfs *** [ 6.251420] *** VALIDATE nfs4 *** [ 6.252955] pstore: using deflate compression [ 6.256866] Platform Keyring initialized [ 6.368329] NET: Registered protocol family 38 [ 6.369952] Key type asymmetric registered [ 6.371642] Asymmetric key parser 'x509' registered [ 6.373757] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 6.377256] io scheduler mq-deadline registered [ 6.378975] io scheduler kyber registered [ 6.380895] io scheduler bfq registered [ 6.382804] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 6.386351] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 6.389427] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 6.392457] ACPI: Power Button [PWRF] [ 6.398203] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 6.405605] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 6.573417] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 6.582348] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 6.719839] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 7.100769] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 7.141090] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 7.146733] Non-volatile memory driver v1.3 [ 7.148732] Linux agpgart interface v0.103 [ 7.194096] virtio_blk virtio1: [vda] 145192 512-byte logical blocks (74.3 MB/70.9 MiB) [ 7.197225] vda: detected capacity change from 0 to 74338304 [ 7.484263] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 7.487255] vdb: detected capacity change from 0 to 1073741824 [ 7.503876] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 7.507326] vdc: detected capacity change from 0 to 2621440000 [ 8.058423] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 8.066384] vdd: detected capacity change from 0 to 2621440000 [ 8.203750] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 8.208371] vde: detected capacity change from 0 to 4294967296 [ 8.238411] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 8.243086] vdf: detected capacity change from 0 to 4294967296 [ 8.254162] libphy: Fixed MDIO Bus: probed [ 8.260435] usbcore: registered new interface driver usbserial_generic [ 8.263834] usbserial: USB Serial support registered for generic [ 8.266713] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 8.272262] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 8.274472] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 8.277600] mousedev: PS/2 mouse device common for all mice [ 8.281480] rtc_cmos 00:05: RTC can wake from S4 [ 8.285574] rtc_cmos 00:05: registered as rtc0 [ 8.288327] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 8.292147] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 8.293223] intel_pstate: CPU model not supported [ 8.299498] hid: raw HID events driver (C) Jiri Kosina [ 8.306907] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 8.307887] usbcore: registered new interface driver usbhid [ 8.316698] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 8.321500] usbhid: USB HID core driver [ 8.324332] drop_monitor: Initializing network drop monitor service [ 8.334260] Initializing XFRM netlink socket [ 8.336975] NET: Registered protocol family 10 [ 8.349336] Segment Routing with IPv6 [ 8.351258] NET: Registered protocol family 17 [ 8.355878] mpls_gso: MPLS GSO support [ 8.372460] RAS: Correctable Errors collector initialized. [ 8.381263] AVX version of gcm_enc/dec engaged. [ 8.386366] AES CTR mode by8 optimization enabled [ 8.541265] sched_clock: Marking stable (8541247253, 0)->(10511539496, -1970292243) [ 8.547878] registered taskstats version 1 [ 8.550436] Loading compiled-in X.509 certificates [ 8.552683] zswap: loaded using pool lzo/zbud [ 8.585898] Key type big_key registered [ 8.605875] Key type encrypted registered [ 8.607956] ima: No TPM chip found, activating TPM-bypass! [ 8.610469] ima: Allocated hash algorithm: sha1 [ 8.612494] ima: No architecture policies found [ 8.614442] evm: Initialising EVM extended attributes: [ 8.616694] evm: security.selinux [ 8.618236] evm: security.ima [ 8.619637] evm: security.capability [ 8.621215] evm: HMAC attrs: 0x1 [ 8.623893] rtc_cmos 00:05: setting system clock to 2026-07-04 05:04:58 UTC (1783141498) [ 8.631684] debug: unmapping init [mem 0xffffffff8dc03000-0xffffffff8ddfffff] [ 8.635635] debug: unmapping init [mem 0xffffffff8c982000-0xffffffff8cc58fff] [ 8.644120] Write protecting the kernel read-only data: 28672k [ 8.648373] debug: unmapping init [mem 0xffffffff8b003000-0xffffffff8b1fffff] [ 8.651929] debug: unmapping init [mem 0xffffffff8b914000-0xffffffff8b9fffff] [ 8.694387] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 8.705292] systemd[1]: Detected virtualization kvm. [ 8.707791] systemd[1]: Detected architecture x86-64. [ 8.710173] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 8.736825] systemd[1]: No hostname configured. [ 8.738466] systemd[1]: Set hostname to . [ 8.740352] random: systemd: uninitialized urandom read (16 bytes read) [ 8.742549] systemd[1]: Initializing machine ID from random generator. [ 8.907885] random: ln: uninitialized urandom read (6 bytes read) [ 9.260160] random: systemd: uninitialized urandom read (16 bytes read) [ 9.267915] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 9.281537] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 9.299957] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Swap. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... Starting Setup Virtual Console... Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Sockets. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 11.506939] device-mapper: uevent: version 1.0.3 [ 11.511687] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook.[ 11.931526] random: fast init done Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted Kernel Configuration File System. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 14.044101] virtio_net virtio0 ens2: renamed from eth0 [ 20.790536] random: crng init done [ 20.794231] random: 7 urandom warning(s) missed due to ratelimiting [ 21.664304] scsi host0: ata_piix [ 21.682180] scsi host1: ata_piix [ 21.690867] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 21.693708] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 24.893023] hrtimer: interrupt took 2504252 ns [ 26.202366] dracut-initqueue[576]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 28.511308] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 32.720495] printk: systemd: 26 output lines suppressed due to ratelimiting [ 34.156513] SELinux: Disabled at runtime. [ 34.342468] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 34.381263] systemd[1]: Detected virtualization kvm. [ 34.392651] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 36.696818] systemd[1]: initrd-switch-root.service: Succeeded. [ 36.711228] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 36.737574] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 36.753047] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 36.759604] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 36.793451] systemd[1]: Starting Journal Service... Starting Journal Service... [ 36.832821] systemd[1]: Activating swap /dev/disk/by-label/SWAP... Activating swap /dev/disk/by-label/SWAP... Starting Remount Root and Kernel File Systems... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-getty.slice. [ 37.017854] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on RPCbind Server Activation Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Control Socket. Mounting Huge Pages File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Switch Root. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Mounting Kernel Debug File System... [ OK ] Stopped target Initrd File Systems. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on Process Core Dump Socket. Starting Apply Kernel Variables... [ OK ] Stopped target Initrd Root File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Reached target Swap. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 38.833100] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 40.432868] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 40.551126] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 41.200451] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 41.350802] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ ***] A start job is running for Configur…only root support (11s / no limit) [ **] A start job is running for Configur…only root support (11s / no limit) [ *] A start job is running for Configur…only root support (12s / no limit)[ 48.684985] Key type dns_resolver registered [ **] A start job is running for Configur…only root support (12s / no limit) [ ***] A start job is running for Configur…only root support (13s / no limit)[ 49.785359] NFS: Registering the id_resolver key type [ 49.790291] Key type id_resolver registered [ 49.793581] Key type id_legacy registered [ *** ] A start job is running for Configur…only root support (13s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg333-server login: [ 114.728240] libcfs: loading out-of-tree module taints kernel. [ 114.777279] Key type ._llcrypt registered [ 114.779469] Key type .llcrypt registered [ 114.881651] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 132.125938] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 133.851807] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 133.873310] alg: No test for adler32 (adler32-zlib) [ 135.237699] Lustre: Lustre: Build Version: 2.17.54_86_gcbe30f7 [ 135.874790] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 137.600269] Key type lgssc registered [ 139.317190] Lustre: Echo OBD driver; http://www.lustre.org/ [ 158.207268] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 198.555393] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 210.589323] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 210.630659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 211.981202] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 212.055610] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 212.186865] Lustre: lustre-MDT0000: new disk, initializing [ 212.355537] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 212.388793] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 216.780393] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 230.671905] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 230.815263] Lustre: 6502:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 230.917022] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 230.922409] Lustre: Skipped 1 previous similar message [ 231.066473] Lustre: lustre-MDT0001: new disk, initializing [ 231.177675] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 231.224894] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 231.238418] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 235.931153] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 240.719559] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 250.457258] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 250.798346] Lustre: lustre-OST0000: new disk, initializing [ 250.811172] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 250.821378] Lustre: 8441:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 250.894788] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 252.637215] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 252.649938] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 252.776779] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 257.723574] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 272.248963] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 272.386395] Lustre: lustre-OST0001: new disk, initializing [ 272.394597] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 272.407790] Lustre: 9514:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 272.484985] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 279.594498] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 279.603397] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 279.673992] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 280.455394] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 293.289588] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 305.182714] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 312.023249] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing check_logdir /tmp/testlogs/ [ 317.442128] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing yml_node [ 322.962356] Lustre: DEBUG MARKER: Client: 2.17.54.86 [ 325.683275] Lustre: DEBUG MARKER: MDS: 2.17.54.86 [ 328.397422] Lustre: DEBUG MARKER: OSS: 2.17.54.86 [ 329.913161] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Jul 4 01:10:18 EDT 2026 [ 346.511272] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 347.716324] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 364.512925] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 364.520197] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 364.542368] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 366.563349] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 366.564146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 366.582152] Lustre: Skipped 1 previous similar message [ 366.592562] Lustre: Skipped 3 previous similar messages [ 369.616232] Lustre: server umount lustre-MDT0000 complete [ 373.510664] LustreError: 8440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783141863 with bad export cookie 11644369241058531370 [ 373.513236] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 373.522679] LustreError: 8440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 376.814424] LustreError: 6509:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 376.820479] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 376.836343] LustreError: 6509:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 376.836884] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 376.869067] Lustre: Skipped 2 previous similar messages [ 378.861571] LustreError: 6508:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 378.861815] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 378.880358] LustreError: 6508:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 378.884735] Lustre: Skipped 2 previous similar messages [ 380.033062] Lustre: server umount lustre-MDT0001 complete [ 384.152207] Lustre: server umount lustre-OST0000 complete [ 388.047775] Lustre: server umount lustre-OST0001 complete [ 393.118935] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 402.060332] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 436.871543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 437.128906] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 437.170156] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 437.268218] Lustre: lustre-MDT0000: new disk, initializing [ 437.343430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 437.361229] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 441.518388] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 451.866622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 452.007177] Lustre: 15067:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 452.085327] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 452.089577] Lustre: Skipped 1 previous similar message [ 452.175311] Lustre: lustre-MDT0001: new disk, initializing [ 452.275844] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 452.341316] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 452.361691] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 456.467748] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 465.479675] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 470.688394] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 476.726175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 476.996888] Lustre: lustre-OST0000: new disk, initializing [ 476.999683] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 477.008070] Lustre: 17035:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 477.089606] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 478.173755] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 478.189094] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 478.285171] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 482.393825] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 490.681563] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 494.775496] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 494.958939] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 499.432847] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 499.635700] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 503.786422] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 503.795540] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 503.807809] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 503.813186] Lustre: Skipped 1 previous similar message [ 507.380464] Lustre: server umount lustre-OST0000 complete [ 514.020504] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 514.022237] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 514.023753] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 514.035648] Lustre: Skipped 2 previous similar messages [ 514.066349] LustreError: Skipped 1 previous similar message [ 516.611445] Lustre: server umount lustre-MDT0000 complete [ 519.141958] LustreError: 16438:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 519.152773] LustreError: 16438:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 520.185565] LustreError: 15058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783142010 with bad export cookie 11644369241058532903 [ 520.202422] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 520.203447] LustreError: 15058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 520.571586] Lustre: server umount lustre-MDT0001 complete [ 525.848791] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 01:13:34 (1783142014) [ 544.123736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 544.684104] LustreError: 19909:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 544.767288] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 548.671346] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 549.861935] LustreError: 19910:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 556.594076] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 561.805389] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 569.532754] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 575.208574] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 577.460276] Lustre: Failing over lustre-MDT0000 [ 577.528214] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 577.536363] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 577.560177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 577.564690] Lustre: Skipped 1 previous similar message [ 577.811485] Lustre: server umount lustre-MDT0000 complete [ 579.555216] LustreError: 19910:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 579.574962] LustreError: 19910:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 595.745100] Lustre: 3636:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783142069/real 1783142069] req@ffff9e9e800a4380 x1869759512737920/t0(0) o400->MGC192.168.203.133@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1783142085 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 595.773227] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 595.787971] LustreError: 20912:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 595.813723] LustreError: 20912:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 596.868539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 605.404343] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 609.698506] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 610.785034] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 610.789888] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 610.820604] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 618.622951] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 626.148352] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 626.152793] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 626.163693] Lustre: Skipped 1 previous similar message [ 626.181663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 626.192725] Lustre: Skipped 1 previous similar message [ 629.078728] Lustre: server umount lustre-MDT0000 complete [ 632.291401] LustreError: 19910:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 632.312172] LustreError: 19910:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 633.139848] LustreError: 20643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783142123 with bad export cookie 11644369241058534317 [ 633.152742] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 633.162049] LustreError: 20643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 633.578854] Lustre: server umount lustre-MDT0001 complete [ 642.152808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 642.677404] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 642.683129] Lustre: Skipped 2 previous similar messages [ 646.555583] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 655.251731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 659.968295] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 667.512968] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 672.916657] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 675.292140] Lustre: Failing over lustre-MDT0000 [ 675.633248] Lustre: server umount lustre-MDT0000 complete [ 676.321825] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 676.336451] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 694.649830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 694.752599] Lustre: 3634:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783142168/real 1783142168] req@ffff9e9d473a8e00 x1869759512767104/t0(0) o400->MGC192.168.203.133@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1783142184 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 694.792481] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 694.812577] Lustre: MGS: Not available for connect from 0@lo (not set up) [ 694.823134] Lustre: Skipped 2 previous similar messages [ 695.166590] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 699.029556] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 700.271294] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 700.283651] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 700.292943] Lustre: Skipped 1 previous similar message [ 700.307237] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 707.894787] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 718.502651] Lustre: server umount lustre-MDT0000 complete [ 719.849669] LustreError: 23250:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 719.876717] LustreError: 23250:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 15 previous similar messages [ 722.624862] LustreError: 23232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783142212 with bad export cookie 11644369241058535255 [ 722.641294] LustreError: 23232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 722.641551] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 723.207636] Lustre: server umount lustre-MDT0001 complete [ 732.557819] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 01:17:00 (1783142220) [ 749.337052] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 760.353897] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 799.184804] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 809.210324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 809.437866] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 809.469961] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 809.554265] Lustre: lustre-MDT0000: new disk, initializing [ 809.644323] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 809.652655] Lustre: Skipped 2 previous similar messages [ 809.665717] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 813.626702] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 822.523581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 822.647883] Lustre: 29624:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 822.878872] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 826.652176] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 835.082422] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 840.369478] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 846.747748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 847.112163] Lustre: 31592:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 848.883600] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 848.923485] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 848.959121] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 848.985366] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 852.821434] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 862.122856] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 867.952231] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 889.838696] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 889.842753] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 889.854503] Lustre: Skipped 3 previous similar messages [ 889.873902] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 889.884252] Lustre: Skipped 3 previous similar messages [ 894.796410] Lustre: server umount lustre-OST0000 complete [ 904.000284] Lustre: server umount lustre-MDT0000 complete [ 905.189084] LustreError: 29635:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 905.211709] LustreError: 29635:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 907.124119] LustreError: 29617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783142396 with bad export cookie 11644369241058535661 [ 907.137221] LustreError: 29617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 907.139067] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 907.521588] Lustre: server umount lustre-MDT0001 complete [ 917.654279] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 920.024670] Key type lgssc unregistered [ 920.292629] LNet: 33567:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 920.310796] LNetError: 33567:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 920.328149] LNet: Removed LNI 192.168.203.133@tcp [ 921.236159] Key type .llcrypt unregistered [ 921.241225] Key type ._llcrypt unregistered [ 943.394015] Key type ._llcrypt registered [ 943.396399] Key type .llcrypt registered [ 943.510385] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 944.264977] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 944.343421] alg: No test for adler32 (adler32-zlib) [ 945.395670] Lustre: Lustre: Build Version: 2.17.54_86_gcbe30f7 [ 945.655301] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 947.336227] Key type lgssc registered [ 948.259455] Lustre: Echo OBD driver; http://www.lustre.org/ [ 954.961435] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 954.974814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 956.335400] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 959.911507] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 966.628082] LustreError: 34983:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 966.642958] LustreError: 34983:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 966.758298] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 967.059842] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 970.148920] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 975.717664] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 980.482923] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 985.683384] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 985.944808] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 991.089177] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 992.120580] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 998.015710] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1001.350691] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1014.275392] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 01:21:42 (1783142502) [ 1015.623280] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1017.122591] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1022.760364] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 01:21:51 (1783142511) [ 1029.584955] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 01:21:57 (1783142517) [ 1030.450836] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1038.241976] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1038.251182] Lustre: Skipped 2 previous similar messages [ 1039.877795] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 01:22:08 (1783142528) [ 1043.933576] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1043.939050] Lustre: Skipped 3 previous similar messages [ 1059.758920] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1059.768512] Lustre: Skipped 6 previous similar messages [ 1062.317513] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 01:22:30 (1783142550) [ 1064.989272] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1067.941547] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1075.482597] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 01:22:43 (1783142563) [ 1077.028148] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1077.039564] Lustre: Skipped 3 previous similar messages [ 1088.724895] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 01:22:56 (1783142576) [ 1103.844995] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1103.849525] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1103.860055] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1107.418422] Lustre: server umount lustre-OST0000 complete [ 1110.496972] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1110.502719] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1110.514579] Lustre: Skipped 1 previous similar message [ 1110.517489] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1110.520321] Lustre: Skipped 1 previous similar message [ 1114.086310] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1114.089049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1114.115355] Lustre: Skipped 1 previous similar message [ 1119.201851] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1119.207799] Lustre: Skipped 1 previous similar message [ 1123.299609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1123.308093] Lustre: Skipped 1 previous similar message [ 1124.320157] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1124.428420] LustreError: 42092:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x2:0x0].0x0 (00000000c998ec0d) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 1124.459888] Lustre: server umount lustre-MDT0000 complete [ 1127.346540] LustreError: 34962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783142617 with bad export cookie 17833177266184833693 [ 1127.349535] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1127.355611] LustreError: 34962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1127.590137] Lustre: server umount lustre-MDT0001 complete [ 1136.871590] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 1138.927392] Key type lgssc unregistered [ 1139.175503] LNet: 42844:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1139.187969] LNetError: 42844:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1140.262507] LNet: Removed LNI 192.168.203.133@tcp [ 1140.896323] Key type .llcrypt unregistered [ 1140.898601] Key type ._llcrypt unregistered [ 1154.345941] Key type ._llcrypt registered [ 1154.347115] Key type .llcrypt registered [ 1154.412150] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 1165.260967] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 1165.820347] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1165.847738] alg: No test for adler32 (adler32-zlib) [ 1166.779916] Lustre: Lustre: Build Version: 2.17.54_86_gcbe30f7 [ 1166.957965] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 1168.592184] Key type lgssc registered [ 1169.448601] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1194.930370] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1194.953405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1196.105566] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1196.134079] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1196.222612] Lustre: lustre-MDT0000: new disk, initializing [ 1196.282108] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1196.298612] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1199.462521] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1206.954475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1207.068852] Lustre: 45973:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1207.119850] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1207.125259] Lustre: Skipped 1 previous similar message [ 1207.196554] Lustre: lustre-MDT0001: new disk, initializing [ 1207.267842] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1207.303412] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1207.309148] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1210.713820] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1218.078340] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1221.977925] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1226.438506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1226.612704] Lustre: lustre-OST0000: new disk, initializing [ 1226.620748] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1226.627689] Lustre: 47942:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1226.706341] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1231.166055] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1234.977938] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1234.983452] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1235.079974] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1238.268770] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1241.532973] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1241.659924] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1244.884597] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1245.071700] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1250.278267] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1250.279171] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1250.298618] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1250.305419] Lustre: Skipped 1 previous similar message [ 1252.677309] Lustre: server umount lustre-OST0000 complete [ 1255.406570] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1255.409257] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1255.421507] Lustre: Skipped 2 previous similar messages [ 1260.516959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1260.521546] Lustre: Skipped 2 previous similar messages [ 1261.671342] Lustre: server umount lustre-MDT0000 complete [ 1264.451359] LustreError: 45965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783142754 with bad export cookie 9577559085876219099 [ 1264.455266] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1264.462685] LustreError: 45965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1264.695824] Lustre: server umount lustre-MDT0001 complete [ 1271.771728] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 01:26:00 (1783142760) [ 1276.522227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1276.945090] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1280.298131] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1282.018446] LustreError: 50401:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1282.032602] LustreError: 50401:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1286.704441] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1286.975900] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1290.175735] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1295.682641] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1299.671873] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1304.521329] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1304.783343] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1309.066151] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1313.498153] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1326.546464] LustreError: 52950:0:(llog.c:444:llog_init_handle()) llog type is not specified! [ 1327.207866] Lustre: Modifying parameter general.timeout=1 in log params [ 1328.748796] Lustre: Modifying parameter general.timeout=3 in log params [ 1328.752872] Lustre: Skipped 1 previous similar message [ 1330.778169] Lustre: Modifying parameter general.timeout=6 in log params [ 1330.782399] Lustre: Skipped 2 previous similar messages [ 1335.301083] Lustre: Modifying parameter general.timeout=13 in log params [ 1335.304052] Lustre: Skipped 6 previous similar messages [ 1343.556370] Lustre: Modifying parameter general.timeout=26 in log params [ 1343.560048] Lustre: Skipped 12 previous similar messages [ 1360.127334] Lustre: Modifying parameter general.timeout=54 in log params [ 1360.130733] Lustre: Skipped 27 previous similar messages [ 1392.269382] Lustre: Modifying parameter general.timeout=109 in log params [ 1392.273088] Lustre: Skipped 54 previous similar messages [ 1409.762430] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 01:28:18 (1783142898) [ 1411.524342] Lustre: 59870:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1412.543365] Lustre: 59968:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1414.755635] Lustre: 60165:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1414.762245] Lustre: 60165:0:(mgs_llog.c:1447:mgs_modify_param()) Skipped 1 previous similar message [ 1433.570911] Lustre: 61714:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1433.575712] Lustre: 61714:0:(mgs_llog.c:1447:mgs_modify_param()) Skipped 2 previous similar messages [ 1433.591931] LustreError: 61714:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1436.402386] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 01:28:44 (1783142924) [ 1444.920458] Lustre: server umount lustre-MDT0000 complete [ 1445.857128] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1445.862826] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1445.874101] LustreError: 50401:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1446.882032] LustreError: 50400:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1446.897049] LustreError: 50400:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1446.932122] LustreError: 53445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783142936 with bad export cookie 9577559085876219974 [ 1446.935146] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1446.942571] LustreError: 53445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1447.128273] Lustre: server umount lustre-MDT0001 complete [ 1459.702222] Lustre: server umount lustre-OST0000 complete [ 1476.967713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1477.183793] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1477.206605] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1477.213835] Lustre: Skipped 43 previous similar messages [ 1477.382782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1479.867984] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1485.323320] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1485.414996] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1485.430711] Lustre: 64243:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1488.103712] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1492.551770] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1496.103408] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1499.805608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1499.914635] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1503.798818] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1508.483560] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1509.634702] Lustre: 66807:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1518.056065] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1518.056564] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1518.058414] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1518.063369] LustreError: Skipped 1 previous similar message [ 1518.077738] Lustre: Skipped 3 previous similar messages [ 1519.659787] Lustre: server umount lustre-OST0000 complete [ 1521.633203] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1523.172154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1523.173298] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1523.174904] Lustre: Skipped 2 previous similar messages [ 1523.189658] Lustre: Skipped 1 previous similar message [ 1527.702864] Lustre: server umount lustre-MDT0000 complete [ 1528.290706] LustreError: 64250:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1528.309825] LustreError: 64250:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1529.876159] LustreError: 64235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783143019 with bad export cookie 9577559085876222620 [ 1529.877710] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1529.886810] LustreError: 64235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1530.087421] Lustre: server umount lustre-MDT0001 complete [ 1535.514977] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 1537.106470] Key type lgssc unregistered [ 1537.287846] LNet: 68013:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1537.292201] LNetError: 68013:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1537.300501] LNet: Removed LNI 192.168.203.133@tcp [ 1537.825151] Key type .llcrypt unregistered [ 1537.827186] Key type ._llcrypt unregistered [ 1552.759303] Key type ._llcrypt registered [ 1552.761253] Key type .llcrypt registered [ 1552.837362] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 1553.289328] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1553.342033] alg: No test for adler32 (adler32-zlib) [ 1554.295364] Lustre: Lustre: Build Version: 2.17.54_86_gcbe30f7 [ 1554.420286] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 1556.024419] Key type lgssc registered [ 1556.656582] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1561.136892] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1561.149135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1562.407618] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1564.623342] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1569.812940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1570.167234] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1572.526494] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1577.076083] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1580.278630] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1584.459550] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1584.772742] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1588.348988] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1592.568278] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1598.512376] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1605.605785] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1605.607502] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1605.614478] Lustre: Skipped 2 previous similar messages [ 1605.620277] Lustre: Skipped 1 previous similar message [ 1609.038124] LustreError: 72442:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x5:0x0].0x0 (000000006c0045cc) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 1609.092064] Lustre: server umount lustre-MDT0000 complete [ 1610.726228] LustreError: 70136:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1610.746124] LustreError: 70136:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1610.990900] LustreError: 69388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783143100 with bad export cookie 15547370002788076951 [ 1610.996546] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1611.001437] LustreError: 69388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1611.173420] Lustre: server umount lustre-MDT0001 complete [ 1623.610266] Lustre: server umount lustre-OST0000 complete [ 1636.879525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1637.015923] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1637.045481] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1637.203593] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1639.148684] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1643.437849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1643.504345] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1643.522590] Lustre: 74270:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1643.790031] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1645.941293] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1649.810852] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1652.840947] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1656.647813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1656.776695] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1656.991252] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1660.112796] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1664.069795] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1670.160950] Lustre: 76846:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1671.184306] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1676.263373] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1676.264235] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1676.265431] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1676.265436] Lustre: Skipped 1 previous similar message [ 1676.273458] Lustre: Skipped 1 previous similar message [ 1676.290315] LustreError: Skipped 1 previous similar message [ 1679.534515] Lustre: server umount lustre-OST0000 complete [ 1684.961827] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1684.969986] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1684.979629] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1684.984084] Lustre: Skipped 1 previous similar message [ 1687.889936] Lustre: server umount lustre-MDT0000 complete [ 1689.996537] LustreError: 75014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783143179 with bad export cookie 15547370002788077784 [ 1690.001036] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1690.004572] LustreError: 75014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1690.199325] Lustre: server umount lustre-MDT0001 complete [ 1696.187520] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 1697.662806] Key type lgssc unregistered [ 1697.835662] LNet: 78052:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1697.842247] LNetError: 78052:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1697.851879] LNet: Removed LNI 192.168.203.133@tcp [ 1698.322647] Key type .llcrypt unregistered [ 1698.324281] Key type ._llcrypt unregistered [ 1711.617812] Key type ._llcrypt registered [ 1711.619422] Key type .llcrypt registered [ 1711.672732] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 1712.257337] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1712.274584] alg: No test for adler32 (adler32-zlib) [ 1713.220194] Lustre: Lustre: Build Version: 2.17.54_86_gcbe30f7 [ 1713.355944] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 1714.960198] Key type lgssc registered [ 1715.500548] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1719.783711] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1719.798234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1721.056022] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1723.226544] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1728.055901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1728.404167] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1730.617360] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1734.421772] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1737.268347] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1741.096692] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1741.440275] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1744.561987] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1748.147378] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1755.445848] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1756.286455] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1756.290444] Lustre: Skipped 1 previous similar message [ 1757.594100] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1757.597770] Lustre: Skipped 2 previous similar messages [ 1759.839199] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1759.843388] Lustre: Skipped 4 previous similar messages [ 1764.017827] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1764.020666] Lustre: Skipped 9 previous similar messages [ 1772.389350] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1772.395650] Lustre: Skipped 17 previous similar messages [ 1788.793675] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1788.799811] Lustre: Skipped 36 previous similar messages [ 1803.527733] Lustre: 87513:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 1805.061576] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 01:34:53 (1783143293) [ 1805.960638] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 1806.988401] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 01:34:55 (1783143295) [ 1810.407874] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1810.408519] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1810.416977] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1810.421660] Lustre: Skipped 1 previous similar message [ 1815.523300] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1815.526257] Lustre: Skipped 2 previous similar messages [ 1820.646287] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1820.650468] Lustre: Skipped 1 previous similar message [ 1823.200125] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1823.278734] Lustre: server umount lustre-OST0000 complete [ 1825.761480] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1825.761838] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1825.766127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1825.770848] Lustre: Skipped 1 previous similar message [ 1830.884996] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1830.889457] Lustre: Skipped 2 previous similar messages [ 1831.355673] Lustre: server umount lustre-MDT0000 complete [ 1833.123594] LustreError: 79425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783143322 with bad export cookie 16968277117395821201 [ 1833.125465] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1833.130304] LustreError: 79425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1833.300478] Lustre: server umount lustre-MDT0001 complete [ 1838.352272] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 1839.689181] Key type lgssc unregistered [ 1839.848417] LNet: 88761:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1839.853372] LNetError: 88761:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1839.863588] LNet: Removed LNI 192.168.203.133@tcp [ 1840.260322] Key type .llcrypt unregistered [ 1840.261759] Key type ._llcrypt unregistered [ 1842.056112] Key type ._llcrypt registered [ 1842.057935] Key type .llcrypt registered [ 1842.118820] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 1844.282967] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules [ 1844.730222] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1844.776292] alg: No test for adler32 (adler32-zlib) [ 1845.689071] Lustre: Lustre: Build Version: 2.17.54_86_gcbe30f7 [ 1845.832925] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 1847.448140] Key type lgssc registered [ 1847.944154] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1848.479130] LustreError: 89915:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 1857.720554] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 1861.985160] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1869.252434] LustreError: 89915:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 1869.263676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1870.536895] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1872.291366] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1876.317602] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 01:36:05 (1783143365) [ 1878.542246] LustreError: 90969:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1878.548423] LustreError: 90969:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1878.554275] LustreError: 90969:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 8, retries 0, failed: rc = -5 [ 1878.718374] Lustre: server umount lustre-MDT0000 complete [ 1884.550590] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 1885.719866] Key type lgssc unregistered [ 1885.865135] LNet: 92461:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1885.868841] LNetError: 92461:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1885.884165] LNet: Removed LNI 192.168.203.133@tcp [ 1886.250515] Key type .llcrypt unregistered [ 1886.252521] Key type ._llcrypt unregistered [ 1895.217584] Key type ._llcrypt registered [ 1895.219448] Key type .llcrypt registered [ 1895.277977] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 1895.837621] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1895.846444] alg: No test for adler32 (adler32-zlib) [ 1896.761210] Lustre: Lustre: Build Version: 2.17.54_86_gcbe30f7 [ 1896.887362] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 1898.488263] Key type lgssc registered [ 1899.063235] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1902.940433] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1902.950748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1904.185498] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1905.994430] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1910.139193] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1910.452842] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1912.413443] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1915.911527] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1918.556747] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1921.799799] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1922.114749] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1924.551675] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1927.435335] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1967.095216] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 01:37:35 (1783143455) [ 1968.125403] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 1969.106363] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 01:37:37 (1783143457) [ 1971.684624] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1971.688433] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1971.693207] Lustre: Skipped 1 previous similar message [ 1971.694220] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1971.699239] Lustre: Skipped 2 previous similar messages [ 1976.803947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1976.807119] Lustre: Skipped 2 previous similar messages [ 1977.084947] Lustre: server umount lustre-MDT0000 complete [ 1978.940458] LustreError: 93642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783143468 with bad export cookie 12244584100435925385 [ 1978.943429] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1978.947599] LustreError: 93642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1979.093499] Lustre: server umount lustre-MDT0001 complete [ 1990.379206] Lustre: server umount lustre-OST0000 complete [ 1996.561184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1996.853311] LustreError: 97948:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1996.870242] LustreError: 97948:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1996.961264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1999.248960] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2002.402954] LustreError: 97949:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2003.969142] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2004.455192] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2006.726373] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2010.216374] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2012.701869] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2018.628484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2018.728158] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2018.732190] LustreError: 97936:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2018.743493] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2018.747778] LustreError: 100030:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2018.751371] LustreError: 100030:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2018.754890] LustreError: 100030:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2018.788814] Lustre: server umount lustre-OST0000 complete [ 2018.791732] LustreError: 100030:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2022.802179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2022.906187] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2022.914349] LustreError: 97938:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2022.919976] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2022.928713] LustreError: 100492:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2022.934828] LustreError: 100492:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2022.938884] LustreError: 100492:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2022.979980] Lustre: server umount lustre-OST0000 complete [ 2022.981894] LustreError: 100492:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2027.706335] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2027.796570] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2027.799395] Lustre: Found index 0 for lustre-OST0000, updating log [ 2027.802483] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2027.851806] Lustre: lustre-OST0000: new disk, initializing [ 2027.854838] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2027.859351] Lustre: 101026:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2027.997565] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2030.764432] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2032.704800] Lustre: Failing over lustre-OST0000 [ 2032.783346] Lustre: server umount lustre-OST0000 complete [ 2035.170075] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2035.170607] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2035.171116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2035.179885] Lustre: Skipped 2 previous similar messages [ 2040.292122] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2040.300286] Lustre: Skipped 2 previous similar messages [ 2045.410864] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2045.414958] Lustre: Skipped 1 previous similar message [ 2048.992139] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2049.136921] Lustre: server umount lustre-MDT0000 complete [ 2050.530514] LustreError: 97948:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2050.543121] LustreError: 97948:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2051.427226] LustreError: 97929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783143541 with bad export cookie 12244584100435928115 [ 2051.433276] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2051.433324] LustreError: 97929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2061.855372] Lustre: server umount lustre-MDT0001 complete [ 2066.738808] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 01:39:15 (1783143555) [ 2075.953650] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 2081.218984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2081.425685] LustreError: 103368:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2081.466988] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2083.619751] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2086.882404] LustreError: 103369:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2087.950281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2090.325287] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2091.716548] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2095.095430] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2095.408557] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2097.450211] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2097.470222] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2097.471100] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2098.506920] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2103.027043] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2103.077434] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2103.154465] Lustre: lustre-OST0001: new disk, initializing [ 2103.158486] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2103.163977] Lustre: 105689:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2106.448014] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2109.452832] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2109.456334] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2109.510412] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2110.922174] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2115.652855] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2118.097446] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.33@tcp (stopping) [ 2119.650935] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2119.660580] Lustre: Skipped 2 previous similar messages [ 2123.702893] Lustre: server umount lustre-MDT0001 complete [ 2124.771558] LustreError: 104098:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2124.783345] LustreError: 104098:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2128.527429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2128.600261] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2128.602808] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2128.634322] Lustre: 103358:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2129.069021] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2129.073620] Lustre: Skipped 3 previous similar messages [ 2130.279366] LustreError: 107307:0:(obd_mount.c:225:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2130.284069] LustreError: 107307:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2130.292321] LustreError: 103360:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2130.346658] LustreError: 107324:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.133@tcp: cfg command failed: rc = -17 [ 2130.351045] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2130.351045] [ 2131.188519] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2134.505865] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2136.293045] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 01:40:25 (1783143625) [ 2144.704260] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 2153.698627] Lustre: 109297:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2164.944415] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2166.720942] Lustre: 110429:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2231.266171] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2231.272122] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2231.279872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2231.283715] Lustre: Skipped 4 previous similar messages [ 2234.064283] Lustre: server umount lustre-MDT0000 complete [ 2236.448757] LustreError: 103349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783143726 with bad export cookie 12244584100435929032 [ 2236.451096] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2236.456044] LustreError: 103349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2236.632316] Lustre: server umount lustre-MDT0001 complete [ 2248.225368] Lustre: server umount lustre-OST0000 complete [ 2260.618692] Lustre: server umount lustre-OST0001 complete [ 2263.831546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2268.768564] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2273.416084] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2278.305416] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2287.648583] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 2293.011594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2293.029140] Lustre: lustre-MDT0000: reset Object Index mappings [ 2293.274719] LustreError: 114377:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2293.282709] LustreError: 114377:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2293.350887] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2295.741269] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2300.395266] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2300.408868] Lustre: lustre-MDT0001: reset Object Index mappings [ 2303.240616] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2305.048849] Lustre: 115524:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2309.210731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2310.702849] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:231 to 0x280000400:257) [ 2313.307696] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2318.845314] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:349 to 0x280000401:386) [ 2318.963292] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2322.985196] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2324.467730] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 2324.475466] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:157 to 0x2c0000401:193) [ 2327.806441] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2329.930361] Lustre: 117399:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2360.885348] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 01:44:09 (1783143849) [ 2391.012717] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2391.018552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2391.030819] Lustre: Skipped 5 previous similar messages [ 2391.032522] Lustre: Skipped 4 previous similar messages [ 2395.607426] Lustre: server umount lustre-MDT0000 complete [ 2396.129746] LustreError: 114377:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2396.141546] LustreError: 114377:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 2397.811800] LustreError: 114357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783143887 with bad export cookie 12244584100435975827 [ 2397.815472] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2397.826942] LustreError: 114357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2401.250914] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2401.259610] Lustre: Skipped 2 previous similar messages [ 2404.143590] Lustre: server umount lustre-MDT0001 complete [ 2412.400744] Lustre: server umount lustre-OST0000 complete [ 2420.817187] Lustre: server umount lustre-OST0001 complete [ 2424.661515] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 2430.277956] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 2451.052065] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2451.179212] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2451.196449] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2451.260095] Lustre: lustre-MDT0000: new disk, initializing [ 2451.297689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2451.301709] Lustre: Skipped 3 previous similar messages [ 2451.309346] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2453.771362] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2457.821087] Lustre: server umount lustre-MDT0000 complete [ 2462.598140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2462.763936] Lustre: Found index 0 for lustre-MDT0000, updating log [ 2462.774994] Lustre: 121661:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2462.782260] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 2464.800470] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2467.246443] Lustre: server umount lustre-MDT0000 complete [ 2471.541955] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 01:46:00 (1783143960) [ 2472.522349] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 2473.560295] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 01:46:02 (1783143962) [ 2482.465731] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 2487.279478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2489.455496] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2493.821168] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2493.866781] Lustre: 123711:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2493.896790] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2493.900019] Lustre: Skipped 1 previous similar message [ 2493.943621] Lustre: lustre-MDT0001: new disk, initializing [ 2494.023227] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2494.027709] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2496.053045] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2498.978599] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 2502.523060] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2502.655393] Lustre: lustre-OST0000: new disk, initializing [ 2502.658228] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2502.661933] Lustre: 125288:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2503.699360] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2503.705508] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2503.767976] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2506.079308] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2512.317651] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2512.396603] Lustre: lustre-OST0001: new disk, initializing [ 2512.399084] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2512.402594] Lustre: 126158:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2513.891757] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2513.897121] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2513.963249] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2515.803434] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2521.754400] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2523.794638] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2739.153183] Lustre: 132543:0:(osd_internal.h:1444:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 2739.157104] Lustre: 132543:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 2739.163065] Lustre: 132543:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 2739.165804] Lustre: 132543:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 2739.172578] Lustre: 132543:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 2739.178828] Lustre: 132543:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 2892.162712] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 01:53:00 (1783144380) [ 2908.132285] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2908.133725] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2908.137375] Lustre: Skipped 3 previous similar messages [ 2908.141995] Lustre: Skipped 7 previous similar messages [ 2910.699642] Lustre: server umount lustre-MDT0000 complete [ 2912.332846] LustreError: 132576:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783144402 with bad export cookie 12244584100436038463 [ 2912.338096] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2912.340081] LustreError: 132576:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2913.248970] LustreError: 126960:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2913.261422] LustreError: 126960:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2918.591476] Lustre: server umount lustre-MDT0001 complete [ 2926.538665] Lustre: server umount lustre-OST0000 complete [ 2934.384138] Lustre: server umount lustre-OST0001 complete [ 2937.167793] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 2941.205948] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 2956.995763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2957.122592] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2957.143386] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2957.191197] Lustre: lustre-MDT0000: new disk, initializing [ 2957.239955] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2957.245033] Lustre: Skipped 5 previous similar messages [ 2957.254514] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2959.239192] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2965.003421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2965.083676] Lustre: 135570:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2965.260814] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2967.026776] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2971.860210] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2974.434612] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2977.605356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2977.760547] Lustre: 137539:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2979.475407] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2979.483029] Lustre: Skipped 1 previous similar message [ 2979.488569] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2979.538050] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2980.566489] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2985.095223] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2989.309411] Lustre: lustre-MDD0000: changelog on [ 2990.532166] Lustre: lustre-MDD0001: changelog on [ 3469.326411] Lustre: lustre-MDD0001: changelog off [ 3470.123629] Lustre: 135577:0:(llog_cat.c:977:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 3470.480732] Lustre: lustre-MDD0000: changelog off [ 3476.450495] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3476.451914] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3476.454584] LustreError: Skipped 1 previous similar message [ 3476.458935] Lustre: Skipped 2 previous similar messages [ 3476.459799] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3476.464583] Lustre: Skipped 5 previous similar messages [ 3478.303553] Lustre: server umount lustre-OST0000 complete [ 3485.841717] Lustre: server umount lustre-MDT0000 complete [ 3486.689535] LustreError: 135582:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3486.698282] LustreError: 135582:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3487.212761] LustreError: 135563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783144977 with bad export cookie 12244584100436847950 [ 3487.217653] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3487.217868] LustreError: 135563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3487.349543] Lustre: server umount lustre-MDT0001 complete [ 3491.326877] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 3492.441434] Key type lgssc unregistered [ 3492.576328] LNet: 146178:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3492.579661] LNetError: 146178:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3492.590567] LNet: Removed LNI 192.168.203.133@tcp [ 3492.940109] Key type .llcrypt unregistered [ 3492.941765] Key type ._llcrypt unregistered [ 3500.511602] Key type ._llcrypt registered [ 3500.513211] Key type .llcrypt registered [ 3500.553723] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 3507.621588] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 3508.034800] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3508.072100] alg: No test for adler32 (adler32-zlib) [ 3508.940832] Lustre: Lustre: Build Version: 2.17.54_86_gcbe30f7 [ 3509.051243] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 3510.640193] Key type lgssc registered [ 3511.070923] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3524.196316] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3524.203501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3525.307167] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3525.321967] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3525.363537] Lustre: lustre-MDT0000: new disk, initializing [ 3525.395741] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3525.405897] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3527.020243] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3531.514900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3531.563824] Lustre: 149353:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3531.591906] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3531.594583] Lustre: Skipped 1 previous similar message [ 3531.632567] Lustre: lustre-MDT0001: new disk, initializing [ 3531.667942] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3531.699670] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3531.704746] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3533.116641] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3537.164750] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3539.027157] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3541.503892] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3541.630276] Lustre: lustre-OST0000: new disk, initializing [ 3541.632816] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3541.636935] Lustre: 151321:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3541.675904] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3543.774083] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3547.596262] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3548.153368] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3548.157618] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3548.189162] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3555.364658] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_module obdecho/obdecho [ 3556.068917] LustreError: 152323:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 3556.072520] LustreError: 152323:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 3556.680928] LustreError: 152419:0:(echo_client.c:1883:echo_md_handler()) Can not resolve the path : rc = -2 [ 3562.465211] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3562.468798] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3562.474273] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3563.489859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3563.491717] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3566.169025] Lustre: server umount lustre-MDT0000 complete [ 3567.433286] LustreError: 149346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783145057 with bad export cookie 15046249997407499869 [ 3567.436694] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3567.436879] LustreError: 149346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3567.560639] Lustre: server umount lustre-MDT0001 complete [ 3579.104259] Lustre: server umount lustre-OST0000 complete [ 3586.242519] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 3589.923857] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 3602.871395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3602.972321] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3602.986033] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3603.024314] Lustre: lustre-MDT0000: new disk, initializing [ 3603.049345] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3603.055118] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3604.483570] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3608.889773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3608.931599] Lustre: 156127:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3608.955249] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3608.957280] Lustre: Skipped 1 previous similar message [ 3608.998737] Lustre: lustre-MDT0001: new disk, initializing [ 3609.035859] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3609.051221] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3609.056522] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3610.505594] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3614.152477] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3615.894421] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3618.156392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3618.254966] Lustre: lustre-OST0000: new disk, initializing [ 3618.256933] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3618.260528] Lustre: 158094:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3618.289654] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3619.131037] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3619.134711] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3619.163462] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3620.157835] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3623.808409] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3625.151440] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3625.237243] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3626.521195] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3626.589239] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3629.538635] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3629.539187] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3629.541554] LustreError: Skipped 1 previous similar message [ 3629.550192] Lustre: Skipped 1 previous similar message [ 3629.551500] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3629.556663] Lustre: Skipped 3 previous similar messages [ 3633.292638] Lustre: server umount lustre-OST0000 complete [ 3634.657340] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3634.657377] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3634.659065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3634.661470] Lustre: Skipped 2 previous similar messages [ 3639.777233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3639.780261] Lustre: Skipped 2 previous similar messages [ 3640.662404] Lustre: server umount lustre-MDT0000 complete [ 3641.965123] LustreError: 156118:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783145131 with bad export cookie 15046249997407500814 [ 3641.967404] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3641.971123] LustreError: 156118:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3642.087851] Lustre: server umount lustre-MDT0001 complete [ 3643.292052] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 02:05:32 (1783145132) [ 3643.797343] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 3644.433067] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 02:05:33 (1783145133) [ 3646.984229] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3647.139545] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3648.411855] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3651.269219] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3652.675696] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3654.992740] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3656.663692] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3658.875093] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3660.897800] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3663.166849] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3676.643555] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3676.644905] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3676.647383] Lustre: Skipped 2 previous similar messages [ 3676.650832] Lustre: Skipped 2 previous similar messages [ 3681.562980] Lustre: server umount lustre-MDT0000 complete [ 3681.761275] LustreError: 160519:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3681.766311] LustreError: 160519:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3682.774119] LustreError: 160481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783145172 with bad export cookie 15046249997407501682 [ 3682.776689] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3682.778172] LustreError: 160481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3683.776300] LustreError: 160496:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3686.784391] LustreError: 162848:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3686.881115] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3691.967956] LustreError: 160495:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3691.974307] LustreError: 160495:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3693.025496] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3693.027752] Lustre: Skipped 6 previous similar messages [ 3697.088530] LustreError: 162848:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3697.094480] LustreError: 162848:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3697.120191] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3697.206390] Lustre: server umount lustre-MDT0001 complete [ 3699.340105] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3703.527829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3703.606286] LustreError: 164359:0:(llog.c:1646:llog_backup()) MGC192.168.203.133@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 3703.610398] Lustre: 164359:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.133@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 3723.746064] LustreError: 164364:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3723.752279] LustreError: 164364:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3740.096203] LustreError: 164364:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3740.104320] LustreError: 164364:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 13 previous similar messages [ 3764.704198] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 3764.705860] Lustre: 164359:0:(mgc_request.c:1901:mgc_process_log()) MGC192.168.203.133@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 3772.897915] LustreError: 164386:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3772.904642] LustreError: 164386:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 21 previous similar messages [ 3805.221822] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3805.224492] Lustre: Skipped 2 previous similar messages [ 3806.562614] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3807.267734] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3809.218784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3827.680716] LustreError: 147578:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000592d2b1d x1869763049527552/t0(0) o250->MGC192.168.203.133@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3829.084574] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3831.349787] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3831.877039] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3832.804986] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3833.513080] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3835.048853] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 1 sec [ 3841.620941] Lustre: server umount lustre-MDT0000 complete [ 3842.796163] LustreError: 160480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783145332 with bad export cookie 15046249997407508157 [ 3842.799902] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3842.801449] LustreError: 160480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3842.908468] Lustre: server umount lustre-MDT0001 complete [ 3863.520913] Lustre: server umount lustre-OST0000 complete [ 3870.749395] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 3874.003128] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 3885.162353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3885.237068] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3885.245502] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3885.281151] Lustre: lustre-MDT0000: new disk, initializing [ 3885.299723] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3885.301497] Lustre: Skipped 1 previous similar message [ 3885.306505] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3886.570801] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3890.556803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3890.597710] Lustre: 169436:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3890.688366] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3891.960508] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3895.494936] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3897.092654] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3899.230337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3899.327883] Lustre: 171402:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3901.270676] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3903.030880] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3903.068534] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3904.826883] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3906.115746] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3906.178331] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3907.321697] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3907.382807] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3908.066244] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3908.066865] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3908.068773] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3908.071658] Lustre: Skipped 1 previous similar message [ 3908.074323] Lustre: Skipped 1 previous similar message [ 3910.029386] Lustre: server umount lustre-OST0000 complete [ 3917.334551] Lustre: server umount lustre-MDT0000 complete [ 3918.305295] LustreError: 169441:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3918.311220] LustreError: 169441:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 42 previous similar messages [ 3918.615430] LustreError: 169429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783145408 with bad export cookie 15046249997407524957 [ 3918.618788] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3918.620773] LustreError: 169429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3918.729620] Lustre: server umount lustre-MDT0001 complete [ 3921.512773] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 02:10:10 (1783145410) [ 3923.649922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3925.066673] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3927.817271] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3929.177264] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3931.376872] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3933.149765] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3935.361532] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3937.359597] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3939.649108] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3953.555837] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 02:10:42 (1783145442) [ 3955.170628] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3955.171194] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3955.175393] Lustre: Skipped 2 previous similar messages [ 3955.179895] LustreError: Skipped 1 previous similar message [ 3956.490718] Lustre: server umount lustre-OST0000 complete [ 3963.726458] Lustre: server umount lustre-MDT0000 complete [ 3964.870185] LustreError: 173787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783145454 with bad export cookie 15046249997407525825 [ 3964.872265] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3964.874644] LustreError: 173787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3964.988132] Lustre: server umount lustre-MDT0001 complete [ 3968.363770] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 3969.333437] Key type lgssc unregistered [ 3969.465306] LNet: 177548:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3969.467678] LNetError: 177548:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3969.478247] LNet: Removed LNI 192.168.203.133@tcp [ 3969.773063] Key type .llcrypt unregistered [ 3969.774734] Key type ._llcrypt unregistered [ 3976.702674] Key type ._llcrypt registered [ 3976.703756] Key type .llcrypt registered [ 3976.735648] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 3977.023867] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3977.050045] alg: No test for adler32 (adler32-zlib) [ 3977.891690] Lustre: Lustre: Build Version: 2.17.54_86_gcbe30f7 [ 3977.977068] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 3979.560168] Key type lgssc registered [ 3979.910886] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3982.391351] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3982.397463] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3997.792366] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4002.912329] LustreError: 178767:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.133@tcp: failed processing log, type 4: rc = -110 [ 4008.032513] LustreError: 178767:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 4008.037488] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 4008.039877] LustreError: MGC192.168.203.133@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 4008.043089] LustreError: 178767:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 4008.046822] LustreError: 178767:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 4008.048898] LustreError: 178767:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 4008.065828] Lustre: server umount lustre-OST0000 complete [ 4008.067161] LustreError: 178767:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 4010.728408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4011.859930] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4013.094946] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4015.679210] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4015.784244] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4017.016465] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4019.191653] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4020.853711] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4022.940681] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4023.004691] LustreError: 181078:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 4023.009092] Lustre: 181078:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.203.133@tcp: invalid local config log lustre-OST0000: rc = -22 [ 4023.043630] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4024.771142] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4026.926654] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4031.460910] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4031.462239] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4031.462511] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4031.465429] Lustre: Skipped 2 previous similar messages [ 4033.693679] Lustre: server umount lustre-MDT0000 complete [ 4034.830184] LustreError: 181104:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783145524 with bad export cookie 10636158109018407691 [ 4034.832940] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4034.833855] LustreError: 181104:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4034.917976] Lustre: server umount lustre-MDT0001 complete [ 4045.264719] Lustre: server umount lustre-OST0000 complete [ 4048.128142] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 4051.163739] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 4061.982575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4062.054846] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4062.063777] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4062.099155] Lustre: lustre-MDT0000: new disk, initializing [ 4062.119351] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4062.126156] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4063.351633] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4067.196937] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4067.227989] Lustre: 184595:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4067.249279] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4067.252215] Lustre: Skipped 1 previous similar message [ 4067.290190] Lustre: lustre-MDT0001: new disk, initializing [ 4067.319443] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4067.332093] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4067.335790] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4068.525576] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4071.845308] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4073.409359] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4075.477751] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4075.562497] Lustre: lustre-OST0000: new disk, initializing [ 4075.564166] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4075.566381] Lustre: 186559:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4075.590139] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4076.596690] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4076.599616] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4076.633084] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4077.284679] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4080.809721] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4082.278962] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4082.356776] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4083.766482] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4083.844673] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4086.754392] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4086.754678] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4086.758908] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4086.760398] LustreError: Skipped 1 previous similar message [ 4086.763556] Lustre: Skipped 3 previous similar messages [ 4090.580058] Lustre: server umount lustre-OST0000 complete [ 4091.873155] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4091.873595] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4091.877433] Lustre: Skipped 2 previous similar messages [ 4096.993058] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4096.994846] Lustre: Skipped 2 previous similar messages [ 4102.113171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4102.116020] Lustre: Skipped 1 previous similar message [ 4106.208076] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4106.263392] Lustre: server umount lustre-MDT0000 complete [ 4107.382235] LustreError: 184587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783145597 with bad export cookie 10636158109018408377 [ 4107.384906] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4107.387338] LustreError: 184587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4107.485977] Lustre: server umount lustre-MDT0001 complete [ 4110.105398] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 02:13:19 (1783145599) [ 4115.676051] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 4116.725827] Key type lgssc unregistered [ 4116.862489] LNet: 189501:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4116.865469] LNetError: 189501:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4116.877493] LNet: Removed LNI 192.168.203.133@tcp [ 4117.225428] Key type .llcrypt unregistered [ 4117.226652] Key type ._llcrypt unregistered [ 4124.096926] Key type ._llcrypt registered [ 4124.098292] Key type .llcrypt registered [ 4124.136091] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 4124.447962] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4124.489474] alg: No test for adler32 (adler32-zlib) [ 4125.336398] Lustre: Lustre: Build Version: 2.17.54_86_gcbe30f7 [ 4125.429866] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 4127.008163] Key type lgssc registered [ 4127.355932] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4130.041970] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4130.048477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4131.176651] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4132.452680] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4135.113530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4135.238672] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4136.629290] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4138.904081] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4140.671658] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4143.241213] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4143.320403] Lustre: *** cfs_fail_loc=131e, val=0*** [ 4143.321543] LustreError: 192570:0:(llog.c:1687:llog_backup()) MGC192.168.203.133@tcp: failed to backup log lustre-OST0000: rc = -28 [ 4143.323916] Lustre: 192570:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.203.133@tcp: can't backup local config lustre-OST0000: rc = -28 [ 4143.361048] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4145.146712] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4147.336506] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4148.006777] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4148.008578] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4148.011474] Lustre: Skipped 1 previous similar message [ 4153.313494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4153.315219] Lustre: Skipped 4 previous similar messages [ 4154.128399] Lustre: server umount lustre-MDT0000 complete [ 4155.243270] LustreError: 190688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783145645 with bad export cookie 3140937239821042832 [ 4155.246880] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4155.248225] LustreError: 190688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4155.353726] Lustre: server umount lustre-MDT0001 complete [ 4166.676233] Lustre: server umount lustre-OST0000 complete [ 4169.421373] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 4172.331927] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 4183.182226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4183.279310] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4183.290930] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4183.324249] Lustre: lustre-MDT0000: new disk, initializing [ 4183.341832] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4183.347678] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4184.627541] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4188.698374] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4188.742121] Lustre: 196086:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4188.760072] Lustre: *** cfs_fail_loc=131e, val=0*** [ 4188.761620] Lustre: Skipped 1 previous similar message [ 4188.763449] LustreError: 196867:0:(llog.c:1687:llog_backup()) MGC192.168.203.133@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 4188.766905] LustreError: 196867:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 4188.769834] Lustre: 196867:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.133@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 4188.778353] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4188.780567] Lustre: Skipped 1 previous similar message [ 4188.813971] Lustre: lustre-MDT0001: new disk, initializing [ 4188.846374] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4188.859051] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4188.864299] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4190.149431] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4193.713524] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4195.413960] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4197.458645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4197.551413] Lustre: *** cfs_fail_loc=131e, val=0*** [ 4197.552968] Lustre: Skipped 1 previous similar message [ 4197.554504] LustreError: 198030:0:(llog.c:1687:llog_backup()) MGC192.168.203.133@tcp: failed to backup log lustre-OST0000: rc = -28 [ 4197.558426] LustreError: 198030:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 4197.561251] Lustre: 198030:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.133@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 4197.565161] Lustre: 198030:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 4197.573787] Lustre: lustre-OST0000: new disk, initializing [ 4197.575705] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4197.578380] Lustre: 198056:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4197.601625] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4199.318059] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4200.562948] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4200.566568] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4200.593939] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4202.727989] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4204.032550] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4204.105792] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4205.373997] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4205.440122] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4206.050410] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4206.050609] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4206.053338] LustreError: Skipped 1 previous similar message [ 4206.057530] Lustre: Skipped 1 previous similar message [ 4206.058099] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4206.061955] Lustre: Skipped 1 previous similar message [ 4211.170773] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4211.173650] Lustre: Skipped 1 previous similar message [ 4216.291200] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4216.293185] Lustre: Skipped 1 previous similar message [ 4220.384107] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4220.430240] Lustre: server umount lustre-OST0000 complete [ 4224.992735] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4224.995077] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4224.998455] Lustre: Skipped 1 previous similar message [ 4224.999899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4227.678428] Lustre: server umount lustre-MDT0000 complete [ 4228.907110] LustreError: 196880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783145718 with bad export cookie 3140937239821043532 [ 4228.910349] LustreError: 196880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4228.910786] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4229.020863] Lustre: server umount lustre-MDT0001 complete [ 4231.873669] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 02:15:21 (1783145721) [ 4236.441375] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 4239.766475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4239.884949] LustreError: 201175:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4239.889612] LustreError: 201175:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4239.896589] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4241.079467] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4243.562554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4244.862306] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4245.773406] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4247.855084] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4249.867564] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4252.627693] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4252.687340] Lustre: lustre-OST0001: new disk, initializing [ 4252.688919] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4252.691280] Lustre: 203479:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4254.793477] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4256.258224] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4256.261903] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4256.294285] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4260.154818] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4267.847940] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4268.439967] Lustre: DEBUG MARKER: ADD OST3 [ 4270.102377] loop: module loaded [ 4270.120681] loop0: detected capacity change from 0 to 204800000 [ 4270.191115] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4270.195403] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4270.200368] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4272.189227] loop0: detected capacity change from 0 to 204800000 [ 4274.087598] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4274.159766] Lustre: lustre-OST0003: new disk, initializing [ 4274.162558] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 4274.166769] Lustre: 205213:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4274.212851] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 4274.216136] Lustre: Skipped 3 previous similar messages [ 4275.216020] Lustre: DEBUG MARKER: STOP OST3 [ 4275.741288] Lustre: *** cfs_fail_loc=2109, val=2*** [ 4275.779219] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 4275.784056] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 4275.797549] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 4275.942823] Lustre: Failing over lustre-OST0003 [ 4275.991617] Lustre: server umount lustre-OST0003 complete [ 4276.481636] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4277.793246] LustreError: 205538:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 4280.356609] Lustre: DEBUG MARKER: START OST3 again [ 4280.779609] LustreError: 202659:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.203.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4280.804319] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4280.806652] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4280.808284] Lustre: Skipped 1 previous similar message [ 4282.476722] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4282.556177] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 4283.938879] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4284.360911] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4284.361936] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4284.362741] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4297.499966] Lustre: Failing over lustre-OST0003 [ 4297.535412] Lustre: server umount lustre-OST0003 complete [ 4298.214271] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4298.218526] LustreError: 202659:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4298.222698] LustreError: 202659:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4303.332535] LustreError: 202659:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4303.338285] LustreError: 202659:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4308.453531] LustreError: 202659:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4308.457557] LustreError: 202659:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4310.496832] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4310.499941] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4310.503516] Lustre: Skipped 1 previous similar message [ 4310.505759] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4310.507804] Lustre: Skipped 2 previous similar messages [ 4315.993772] Lustre: server umount lustre-MDT0000 complete [ 4317.112078] LustreError: 203478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783145806 with bad export cookie 3140937239821044400 [ 4317.115534] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4317.115732] LustreError: 203478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4318.688873] LustreError: 203941:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4318.694656] LustreError: 203941:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4323.372532] Lustre: server umount lustre-MDT0001 complete [ 4324.648941] Lustre: server umount lustre-OST0000 complete [ 4325.896653] Lustre: server umount lustre-OST0001 complete [ 4327.985180] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 4330.281281] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 4341.169813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4341.247863] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4341.258582] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4341.292445] Lustre: lustre-MDT0000: new disk, initializing [ 4341.311677] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4341.314764] Lustre: Skipped 1 previous similar message [ 4341.320372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4342.667370] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4346.577044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4346.619299] Lustre: 210622:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4346.731102] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4347.978184] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4351.408140] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4352.966721] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4355.000826] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4355.097209] Lustre: 212588:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4356.347130] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4356.385282] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4357.054150] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4360.665226] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4362.038462] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4362.111135] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4363.547515] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4363.624580] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4366.818343] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4366.818706] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4366.820804] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4366.823585] Lustre: Skipped 6 previous similar messages [ 4366.826272] Lustre: Skipped 8 previous similar messages [ 4370.319340] Lustre: server umount lustre-OST0000 complete [ 4377.620268] Lustre: server umount lustre-MDT0000 complete [ 4378.830187] LustreError: 211416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783145868 with bad export cookie 3140937239821046654 [ 4378.833352] LustreError: 211416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4378.833771] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4378.950283] Lustre: server umount lustre-MDT0001 complete [ 4380.092322] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 02:17:49 (1783145869) [ 4385.288649] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 4388.407372] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 4399.898776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4399.975802] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4399.986209] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4399.988400] Lustre: Skipped 3 previous similar messages [ 4400.021760] Lustre: lustre-MDT0000: new disk, initializing [ 4400.023008] Lustre: Skipped 2 previous similar messages [ 4400.044767] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4400.046900] Lustre: Skipped 2 previous similar messages [ 4401.280214] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4405.236889] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4405.282755] Lustre: 217186:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4405.395702] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4406.708495] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4410.217604] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4411.944686] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4414.099801] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4414.197085] Lustre: 219157:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4415.642460] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4416.027670] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4419.439175] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4420.803207] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4420.872500] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4422.171578] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4422.239921] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4426.210923] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4426.211694] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4426.214017] LustreError: Skipped 2 previous similar messages [ 4426.218306] Lustre: Skipped 3 previous similar messages [ 4428.941421] Lustre: server umount lustre-OST0000 complete [ 4431.330800] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4431.333485] Lustre: Skipped 7 previous similar messages [ 4436.191606] Lustre: server umount lustre-MDT0000 complete [ 4436.449387] LustreError: 217198:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4436.454888] LustreError: 217198:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4437.383145] LustreError: 217179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783145927 with bad export cookie 3140937239821047536 [ 4437.385446] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4437.387772] LustreError: 217179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4437.494587] Lustre: server umount lustre-MDT0001 complete [ 4440.123495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4441.715482] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4444.550261] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4445.967731] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4448.169865] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4449.783746] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4451.811430] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4453.619013] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4455.789040] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4475.874402] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4475.877079] LustreError: Skipped 2 previous similar messages [ 4478.539323] Lustre: server umount lustre-OST0000 complete [ 4485.852749] Lustre: server umount lustre-MDT0000 complete [ 4486.113852] LustreError: 221387:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4486.120957] LustreError: 221387:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4487.020977] LustreError: 221350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783145976 with bad export cookie 3140937239821048418 [ 4487.023812] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4487.024034] LustreError: 221350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4487.125708] Lustre: server umount lustre-MDT0001 complete [ 4591.001560] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 4592.003913] Key type lgssc unregistered [ 4592.120338] LNet: 225052:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4592.123181] LNetError: 225052:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4592.129505] LNet: Removed LNI 192.168.203.133@tcp [ 4592.476136] Key type .llcrypt unregistered [ 4592.477246] Key type ._llcrypt unregistered [ 4600.425404] Key type ._llcrypt registered [ 4600.426399] Key type .llcrypt registered [ 4600.467623] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 4606.368060] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 4606.726448] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4606.737922] alg: No test for adler32 (adler32-zlib) [ 4607.584120] Lustre: Lustre: Build Version: 2.17.54_86_gcbe30f7 [ 4607.657429] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 4609.232111] Key type lgssc registered [ 4609.595015] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4620.767124] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4620.773035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4621.871038] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4621.882610] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4621.920046] Lustre: lustre-MDT0000: new disk, initializing [ 4621.941574] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4621.949307] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4623.429492] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4627.616185] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4627.658130] Lustre: 228376:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4627.679648] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4627.681732] Lustre: Skipped 1 previous similar message [ 4627.716835] Lustre: lustre-MDT0001: new disk, initializing [ 4627.747607] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4627.758359] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4627.761976] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4628.991309] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4632.524250] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4634.215656] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4636.360869] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4636.452641] Lustre: lustre-OST0000: new disk, initializing [ 4636.454546] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4636.456948] Lustre: 230341:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4636.481351] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4638.301651] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4641.731455] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4643.054556] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4643.318188] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4643.322032] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4643.347649] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4644.144378] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 4645.402702] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4645.471832] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4648.417956] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4648.419744] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4648.420166] LustreError: Skipped 1 previous similar message [ 4648.426608] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4648.431225] Lustre: Skipped 1 previous similar message [ 4652.180838] Lustre: server umount lustre-OST0000 complete [ 4653.538313] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4653.538633] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4653.539410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4653.544910] Lustre: Skipped 2 previous similar messages [ 4658.657080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4658.659155] Lustre: Skipped 2 previous similar messages [ 4659.542806] Lustre: server umount lustre-MDT0000 complete [ 4660.708157] LustreError: 228370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783146150 with bad export cookie 6109655911697449588 [ 4660.710931] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4660.711959] LustreError: 228370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4660.815915] Lustre: server umount lustre-MDT0001 complete [ 4666.198411] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 4669.247132] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4669.370028] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4670.597930] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4673.199164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4674.593103] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4675.533939] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4677.593360] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4677.699503] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4677.702510] Lustre: Skipped 1 previous similar message [ 4679.535338] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4682.118095] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4682.180174] Lustre: lustre-OST0001: new disk, initializing [ 4682.182549] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4682.185549] Lustre: 235580:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4683.957380] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4684.419177] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4684.422894] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4684.455861] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4689.134295] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4693.638488] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4697.054604] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 02:23:06 (1783146186) [ 4699.104935] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4699.108686] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4699.113883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4703.768219] Lustre: server umount lustre-MDT0000 complete [ 4704.916066] LustreError: 233260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783146194 with bad export cookie 6109655911697450456 [ 4704.919723] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4704.920219] LustreError: 233260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4705.029415] Lustre: server umount lustre-MDT0001 complete [ 4716.417238] Lustre: server umount lustre-OST0000 complete [ 4727.897226] Lustre: server umount lustre-OST0001 complete [ 4730.001278] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 4733.310931] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 4743.993121] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4744.085063] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4744.093953] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4744.129933] Lustre: lustre-MDT0000: new disk, initializing [ 4744.149578] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4744.151558] Lustre: Skipped 1 previous similar message [ 4744.156207] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4745.523565] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4749.796230] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4749.842252] Lustre: 239521:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4749.967466] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4751.412038] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4755.274830] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4757.153567] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4759.406843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4759.502513] Lustre: lustre-OST0000: new disk, initializing [ 4759.503822] Lustre: Skipped 1 previous similar message [ 4759.505266] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4759.508061] Lustre: Skipped 2 previous similar messages [ 4759.509754] Lustre: 241488:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4761.146530] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4761.150140] Lustre: Skipped 1 previous similar message [ 4761.151649] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4761.189722] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4761.377307] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4764.807224] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4766.039178] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4766.099064] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4767.289156] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4767.350375] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4771.298832] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4771.298892] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4771.305029] Lustre: Skipped 3 previous similar messages [ 4771.308823] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4771.312384] Lustre: Skipped 5 previous similar messages [ 4774.028458] Lustre: server umount lustre-OST0000 complete [ 4775.905054] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4775.908032] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4781.276134] Lustre: server umount lustre-MDT0000 complete [ 4781.537911] LustreError: 239533:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4781.544083] LustreError: 239533:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4782.489282] LustreError: 240318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783146272 with bad export cookie 6109655911697452549 [ 4782.492565] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4782.494410] LustreError: 240318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4782.624572] Lustre: server umount lustre-MDT0001 complete [ 4785.329404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4785.484730] LustreError: 243702:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4785.507316] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4785.509245] Lustre: Skipped 2 previous similar messages [ 4786.840289] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4789.648769] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4791.127487] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4793.512993] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4795.294937] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4797.470327] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4799.492572] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4802.038601] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4948.449929] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4948.453629] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4948.458342] Lustre: Skipped 1 previous similar message [ 4948.461794] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4948.464435] Lustre: Skipped 3 previous similar messages [ 4950.799575] Lustre: server umount lustre-OST0000 complete [ 4964.833150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4964.835229] Lustre: Skipped 8 previous similar messages [ 4966.368134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4966.435805] Lustre: server umount lustre-MDT0000 complete [ 4967.643202] LustreError: 243685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783146457 with bad export cookie 6109655911697453431 [ 4967.646676] LustreError: 243685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4967.647062] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4967.764050] Lustre: server umount lustre-MDT0001 complete [ 5065.681932] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 5066.680496] Key type lgssc unregistered [ 5066.796350] LNet: 247304:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5066.800222] LNetError: 247304:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5066.812280] LNet: Removed LNI 192.168.203.133@tcp [ 5067.125103] Key type .llcrypt unregistered [ 5067.126681] Key type ._llcrypt unregistered [ 5074.782843] Key type ._llcrypt registered [ 5074.783890] Key type .llcrypt registered [ 5074.818837] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 5080.884716] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 5081.248682] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5081.284344] alg: No test for adler32 (adler32-zlib) [ 5082.138655] Lustre: Lustre: Build Version: 2.17.54_86_gcbe30f7 [ 5082.224305] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 5083.800193] Key type lgssc registered [ 5084.174710] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5099.129730] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 5102.573046] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5102.578171] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5103.661219] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5103.671571] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5103.712209] Lustre: lustre-MDT0000: new disk, initializing [ 5103.732483] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5103.738308] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5105.118353] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5109.239280] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5109.285161] Lustre: 251342:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5109.311924] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5109.313707] Lustre: Skipped 1 previous similar message [ 5109.353329] Lustre: lustre-MDT0001: new disk, initializing [ 5109.385044] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5109.419889] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5109.424496] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5110.754157] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5113.019346] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5115.228216] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5115.325658] Lustre: lustre-OST0000: new disk, initializing [ 5115.327683] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5115.330561] Lustre: 252973:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5115.361637] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5116.924087] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5116.927657] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5116.976050] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5117.304221] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5121.342856] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5121.410945] Lustre: lustre-OST0001: new disk, initializing [ 5121.413975] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5121.418582] Lustre: 253845:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5121.453415] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5123.356583] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5124.227590] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5124.231938] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5124.261920] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5128.690548] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5134.420952] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5139.940303] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5139.941516] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5139.945220] Lustre: Skipped 3 previous similar messages [ 5139.948521] Lustre: Skipped 3 previous similar messages [ 5143.522244] Lustre: server umount lustre-MDT0000 complete [ 5144.735110] LustreError: 253844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783146634 with bad export cookie 1199719264464594043 [ 5144.738634] LustreError: 253844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5144.738914] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5144.857644] Lustre: server umount lustre-MDT0001 complete [ 5156.417509] Lustre: server umount lustre-OST0000 complete [ 5167.899466] Lustre: server umount lustre-OST0001 complete [ 5172.275328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5175.255179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5178.013179] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 5181.178547] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 5187.646370] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 5191.069923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5191.077558] Lustre: lustre-MDT0000: reset Object Index mappings [ 5191.252188] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5192.595187] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5195.385060] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5195.391215] Lustre: lustre-MDT0001: reset Object Index mappings [ 5197.006352] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5197.999351] Lustre: 259981:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5200.187966] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5200.331484] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5200.333726] Lustre: Skipped 1 previous similar message [ 5201.381731] LustreError: 260335:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5201.386088] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5201.386947] LustreError: 260335:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5202.319621] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5205.142729] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5206.253086] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 5207.115377] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5210.112672] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5211.255617] Lustre: 261849:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 5216.224955] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5216.228261] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5216.232562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5218.455103] Lustre: server umount lustre-MDT0000 complete [ 5219.622171] LustreError: 258823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783146709 with bad export cookie 1199719264464600819 [ 5219.625165] LustreError: 258823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5219.625784] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5219.735534] Lustre: server umount lustre-MDT0001 complete [ 5231.042598] Lustre: server umount lustre-OST0000 complete [ 5242.456488] Lustre: server umount lustre-OST0001 complete [ 5248.374528] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 02:32:17 (1783146737) [ 5253.286676] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 5256.225907] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 5267.502357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5267.580518] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5267.590311] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5267.631271] Lustre: lustre-MDT0000: new disk, initializing [ 5267.651923] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5267.655529] Lustre: Skipped 1 previous similar message [ 5267.659990] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5268.975287] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5272.966934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5273.008662] Lustre: 266104:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5273.124954] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5274.365122] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5277.899314] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5279.624544] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5281.814182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5281.912547] Lustre: lustre-OST0000: new disk, initializing [ 5281.914063] Lustre: Skipped 1 previous similar message [ 5281.915572] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5281.918830] Lustre: Skipped 2 previous similar messages [ 5281.920704] Lustre: 268071:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5283.194293] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5283.197427] Lustre: Skipped 1 previous similar message [ 5283.199378] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5283.229685] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5283.775430] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5287.374482] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5288.763544] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5288.836647] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5290.248416] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5290.328554] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5293.538983] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5293.539285] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5293.541599] LustreError: Skipped 1 previous similar message [ 5293.547825] Lustre: Skipped 3 previous similar messages [ 5293.548561] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5293.554257] Lustre: Skipped 4 previous similar messages [ 5298.660260] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5298.663508] Lustre: Skipped 1 previous similar message [ 5303.779131] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5303.781056] Lustre: Skipped 1 previous similar message [ 5305.312132] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5305.366056] Lustre: server umount lustre-OST0000 complete [ 5308.899240] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5308.900029] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5308.905052] Lustre: Skipped 2 previous similar messages [ 5312.739221] Lustre: server umount lustre-MDT0000 complete [ 5313.951964] LustreError: 266897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783146803 with bad export cookie 1199719264464602891 [ 5313.955564] LustreError: 266897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5313.955870] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5314.089331] Lustre: server umount lustre-MDT0001 complete [ 5319.825377] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 5323.155029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5323.277291] LustreError: 271000:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5323.289320] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5323.291153] Lustre: Skipped 2 previous similar messages [ 5324.597527] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5327.265957] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5328.650085] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5329.585844] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5331.650116] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5333.442141] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5335.985752] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5336.044778] Lustre: lustre-OST0001: new disk, initializing [ 5336.046791] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5336.049269] Lustre: 273305:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5337.833507] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5337.981261] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5337.984033] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5338.019382] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5342.071069] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5343.147171] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5344.768695] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 5344.783246] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 5347.808900] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5347.811183] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5347.815052] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5347.816868] Lustre: Skipped 2 previous similar messages [ 5352.473542] Lustre: server umount lustre-MDT0000 complete [ 5353.594355] LustreError: 270983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783146843 with bad export cookie 1199719264464603773 [ 5353.598451] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5353.600545] LustreError: 270983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5353.710448] Lustre: server umount lustre-MDT0001 complete [ 5365.124251] Lustre: server umount lustre-OST0000 complete [ 5376.598939] Lustre: server umount lustre-OST0001 complete [ 5381.616896] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 5384.886678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5385.010120] LustreError: 276352:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5386.250472] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5388.895867] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5389.018426] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5389.020222] Lustre: Skipped 4 previous similar messages [ 5390.260973] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5391.152698] Lustre: 277495:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5393.148073] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5394.275880] LustreError: 277847:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5394.284584] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 5395.028088] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5397.571556] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5398.700750] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 5399.479120] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5402.392604] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5403.461400] Lustre: 279367:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 5404.976901] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 5404.985538] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 5407.695441] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 02:34:56 (1783146896) [ 5419.491661] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5419.493600] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5419.495085] Lustre: Skipped 5 previous similar messages [ 5419.497933] Lustre: Skipped 7 previous similar messages [ 5425.239102] Lustre: server umount lustre-MDT0000 complete [ 5426.363166] LustreError: 276335:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783146916 with bad export cookie 1199719264464610080 [ 5426.366068] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5426.366416] LustreError: 276335:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5429.729021] LustreError: 276349:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5429.733892] LustreError: 276349:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5432.558826] Lustre: server umount lustre-MDT0001 complete [ 5448.160080] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5448.196702] Lustre: server umount lustre-OST0000 complete [ 5455.447643] Lustre: server umount lustre-OST0001 complete [ 5457.364723] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 5460.302700] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 5473.980286] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 5477.074370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5477.154910] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5477.165055] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5477.202367] Lustre: lustre-MDT0000: new disk, initializing [ 5477.224053] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5478.422304] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5482.322817] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5482.362541] Lustre: 283599:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5482.482611] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5483.692731] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5487.811839] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5487.903162] Lustre: 285230:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5488.285180] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5489.658970] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5493.474969] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5495.301802] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5495.335061] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5499.437642] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5503.926136] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 02:36:33 (1783146993) [ 5505.507736] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5505.508830] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5505.510930] Lustre: Skipped 4 previous similar messages [ 5505.512612] Lustre: Skipped 8 previous similar messages [ 5510.686331] Lustre: server umount lustre-MDT0000 complete [ 5511.802965] LustreError: 283590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783147001 with bad export cookie 1199719264464617962 [ 5511.807132] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5511.807954] LustreError: 283590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5511.904490] Lustre: server umount lustre-MDT0001 complete [ 5522.946876] Lustre: server umount lustre-OST0000 complete [ 5533.269444] Lustre: server umount lustre-OST0001 complete [ 5535.771472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5535.906908] LustreError: 288340:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5535.912341] LustreError: 288340:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5535.933960] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5535.935759] Lustre: Skipped 6 previous similar messages [ 5537.128094] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5539.612621] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5540.911093] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5542.999573] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5544.525355] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5546.517322] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5548.288243] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5550.406603] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5555.170531] LustreError: 290757:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5555.174591] LustreError: 290757:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5566.295256] loop1: detected capacity change from 0 to 204800000 [ 5566.394714] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5566.398083] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5566.401638] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5567.957321] loop1: detected capacity change from 0 to 204800000 [ 5569.706211] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5569.734798] LustreError: 288328:0:(mgs_llog.c:673:mgs_find_or_make_fsdb_nolock()) fs17448: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 5569.739801] LustreError: 288328:0:(mgs_llog.c:731:mgs_check_index()) Can't get db for fs17448 [ 5569.742339] LustreError: 288328:0:(mgs_handler.c:517:mgs_target_reg()) fs17448-MDT0001: Error checking target index: rc = -13 [ 5569.745576] LustreError: fs17448-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 5569.750765] LustreError: 292073:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [ 5569.754249] LustreError: 292073:0:(tgt_mount.c:2129:server_put_super()) no obd fs17448-MDT0001 [ 5569.757499] LustreError: 292073:0:(tgt_mount.c:116:server_deregister_mount()) fs17448-MDT0001 not registered [ 5569.769768] Lustre: server umount fs17448-MDT0001 complete [ 5569.771795] LustreError: 292073:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 5572.354140] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5572.384708] Lustre: Setting parameter fs17448-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs17448-MDT0001 [ 5572.389504] Lustre: Skipped 2 previous similar messages [ 5572.405335] Lustre: srv-fs17448-MDT0001: No data found on store. Initialize space. [ 5572.407449] Lustre: Skipped 4 previous similar messages [ 5572.441656] Lustre: fs17448-MDT0001: new disk, initializing [ 5572.443017] Lustre: Skipped 3 previous similar messages [ 5572.473994] Lustre: srv-fs17448-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 5572.476162] Lustre: 292537:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs17448-MDT0001-mdtlov: can't create new llog: rc = -115 [ 5572.479447] LustreError: 292537:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs17448-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 5573.682551] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5576.415485] Lustre: server umount fs17448-MDT0001 complete [ 5580.413650] loop1: detected capacity change from 0 to 204800000 [ 5582.191517] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5582.293989] Lustre: 294105:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs17448-MDT0001-mdtlov: can't create new llog: rc = -115 [ 5582.296623] LustreError: 294105:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs17448-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 5583.545893] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5585.052873] Lustre: server umount fs17448-MDT0001 complete [ 5591.011107] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5591.011783] LustreError: 290169:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5591.012048] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5591.012057] Lustre: Skipped 2 previous similar messages [ 5591.013564] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5591.013571] Lustre: Skipped 7 previous similar messages [ 5591.028330] LustreError: 290169:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 18 previous similar messages [ 5602.272087] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5602.339577] Lustre: server umount lustre-MDT0000 complete [ 5604.863941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5604.900215] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5606.213765] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5608.494504] loop1: detected capacity change from 0 to 204800000 [ 5610.198712] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5610.299176] Lustre: 296548:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs17448-MDT0001-mdtlov: can't create new llog: rc = -115 [ 5610.303144] LustreError: 296548:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs17448-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 5610.470737] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5611.475019] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5612.951191] Lustre: server umount fs17448-MDT0001 complete [ 5620.709438] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5625.104751] Lustre: server umount lustre-OST0000 complete [ 5632.408537] Lustre: server umount lustre-MDT0000 complete [ 5633.524110] LustreError: 289083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783147123 with bad export cookie 1199719264464620636 [ 5633.529698] LustreError: 289083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5633.641424] Lustre: server umount lustre-MDT0001 complete [ 5636.852087] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 5637.726370] Key type lgssc unregistered [ 5637.838321] LNet: 298887:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5637.840644] LNetError: 298887:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5637.852288] LNet: Removed LNI 192.168.203.133@tcp [ 5638.127184] Key type .llcrypt unregistered [ 5638.128199] Key type ._llcrypt unregistered [ 5644.847647] Key type ._llcrypt registered [ 5644.848924] Key type .llcrypt registered [ 5644.882336] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 5645.180118] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5645.209098] alg: No test for adler32 (adler32-zlib) [ 5646.048012] Lustre: Lustre: Build Version: 2.17.54_86_gcbe30f7 [ 5646.125779] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 5647.696126] Key type lgssc registered [ 5648.005132] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5650.280253] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5650.286928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5651.421865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5652.625311] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5655.148581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5655.271096] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5656.477807] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5658.534648] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5660.058453] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5662.009272] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5662.123976] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5663.791811] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5665.189144] LustreError: 301960:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5665.194317] LustreError: 301960:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5665.850152] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5670.371833] LustreError: 301962:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5675.493025] LustreError: 301961:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5681.606391] LustreError: 301960:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5681.611584] LustreError: 301960:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5686.721148] LustreError: 301961:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5686.729938] LustreError: 301961:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5687.211242] loop1: detected capacity change from 0 to 204800000 [ 5687.265684] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5687.269374] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5687.273167] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5688.792221] loop1: detected capacity change from 0 to 204800000 [ 5690.541951] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5690.591264] Lustre: fs1668-OST0000: new disk, initializing [ 5690.592818] Lustre: srv-fs1668-OST0000: No data found on store. Initialize space. [ 5690.594996] Lustre: 303888:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5690.614897] Lustre: fs1668-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5692.408289] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5695.681016] Lustre: server umount fs1668-OST0000 complete [ 5696.961501] LustreError: 301960:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5696.967178] LustreError: 301960:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 5699.394891] loop1: detected capacity change from 0 to 204800000 [ 5701.126262] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5701.183424] Lustre: fs1668-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5702.926388] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5705.515751] loop2: detected capacity change from 0 to 204800000 [ 5705.592849] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5705.596279] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5705.599381] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5707.173051] loop2: detected capacity change from 0 to 204800000 [ 5708.890841] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5708.918188] LustreError: fs1668-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [ 5708.923818] LustreError: fs1668-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 5708.927770] LustreError: 306922:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [ 5708.931051] LustreError: 306922:0:(tgt_mount.c:2129:server_put_super()) no obd fs1668-OST0001 [ 5708.933792] LustreError: 306922:0:(tgt_mount.c:116:server_deregister_mount()) fs1668-OST0001 not registered [ 5708.945692] Lustre: server umount fs1668-OST0001 complete [ 5708.947431] LustreError: 306922:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 5712.029158] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5712.067344] Lustre: fs1668-OST0001: new disk, initializing [ 5712.068854] Lustre: srv-fs1668-OST0001: No data found on store. Initialize space. [ 5712.071196] Lustre: 307475:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5712.090387] Lustre: fs1668-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5713.865416] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5717.081874] Lustre: server umount fs1668-OST0000 complete [ 5717.440801] LustreError: 301962:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5717.447833] LustreError: 301962:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 5719.841550] Lustre: server umount fs1668-OST0001 complete [ 5723.554689] loop1: detected capacity change from 0 to 204800000 [ 5725.248561] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5726.990975] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5729.049667] loop2: detected capacity change from 0 to 204800000 [ 5730.774121] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5730.829995] Lustre: fs1668-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5730.832008] Lustre: Skipped 1 previous similar message [ 5732.548011] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5735.155878] Lustre: server umount fs1668-OST0000 complete [ 5737.904888] Lustre: server umount fs1668-OST0001 complete [ 5742.048756] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5742.050895] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5742.054625] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5743.074380] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5743.075540] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5743.076213] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5747.341227] Lustre: server umount lustre-OST0000 complete [ 5752.800854] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5752.803986] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5752.809362] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5752.810962] Lustre: Skipped 1 previous similar message [ 5754.580794] Lustre: server umount lustre-MDT0000 complete [ 5755.645530] LustreError: 300115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783147245 with bad export cookie 8295694340378855765 [ 5755.647469] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5755.648753] LustreError: 300115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5755.758475] Lustre: server umount lustre-MDT0001 complete [ 5758.895549] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 5759.801586] Key type lgssc unregistered [ 5759.920327] LNet: 313386:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5759.922627] LNetError: 313386:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5759.930302] LNet: Removed LNI 192.168.203.133@tcp [ 5760.220092] Key type .llcrypt unregistered [ 5760.221102] Key type ._llcrypt unregistered [ 5769.455145] Key type ._llcrypt registered [ 5769.456059] Key type .llcrypt registered [ 5769.490834] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 5775.296091] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 5775.570165] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5775.581103] alg: No test for adler32 (adler32-zlib) [ 5776.417269] Lustre: Lustre: Build Version: 2.17.54_86_gcbe30f7 [ 5776.502662] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 5778.072112] Key type lgssc registered [ 5778.376154] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5789.083065] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5789.088644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5790.165417] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5790.175282] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5790.210751] Lustre: lustre-MDT0000: new disk, initializing [ 5790.231372] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5790.238558] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5791.460835] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5795.393064] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5795.441251] Lustre: 317215:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5795.468291] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5795.470889] Lustre: Skipped 1 previous similar message [ 5795.515909] Lustre: lustre-MDT0001: new disk, initializing [ 5795.545405] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5795.556101] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5795.559327] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5796.859996] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5800.436862] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5802.064327] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5804.152915] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5804.252834] Lustre: lustre-OST0000: new disk, initializing [ 5804.254917] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5804.258248] Lustre: 319184:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5804.287797] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5805.431394] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5805.434643] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5805.466148] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5806.054022] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5809.499488] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5810.131874] Lustre: Mounted lustre-client [ 5810.422872] Lustre: Unmounted lustre-client [ 5815.267449] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5815.268153] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5815.271014] Lustre: Skipped 1 previous similar message [ 5819.085367] Lustre: server umount lustre-OST0000 complete [ 5820.385372] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5820.385966] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5820.389259] Lustre: Skipped 1 previous similar message [ 5820.391954] Lustre: Skipped 1 previous similar message [ 5825.505128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5825.507190] Lustre: Skipped 2 previous similar messages [ 5826.329251] Lustre: server umount lustre-MDT0000 complete [ 5827.469150] LustreError: 318009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783147317 with bad export cookie 13640341906482541797 [ 5827.472239] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5827.475047] LustreError: 318009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5827.578220] Lustre: server umount lustre-MDT0001 complete [ 5830.761313] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 5831.657640] Key type lgssc unregistered [ 5868.704099] LNet: 1 local NIs in recovery (showing 1): 192.168.203.133@tcp [ 5869.664284] LNet: 321342:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5869.666448] LNetError: 321342:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5940.326271] LNet: Removed LNI 192.168.203.133@tcp [ 5940.602786] Key type .llcrypt unregistered [ 5940.603807] Key type ._llcrypt unregistered [ 5946.078911] Key type ._llcrypt registered [ 5946.080598] Key type .llcrypt registered [ 5946.114146] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 5952.084202] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 5952.538561] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5952.550239] alg: No test for adler32 (adler32-zlib) [ 5953.394447] Lustre: Lustre: Build Version: 2.17.54_86_gcbe30f7 [ 5953.481701] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 5955.056119] Key type lgssc registered [ 5955.398845] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5964.333159] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 02:44:13 (1783147453) [ 5966.346758] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5966.354552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5967.427991] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5967.439080] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5967.473840] Lustre: lustre-MDT0000: new disk, initializing [ 5967.493194] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5967.499595] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5968.802094] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5972.650623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5972.687989] Lustre: 324537:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5972.709233] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5972.710843] Lustre: Skipped 1 previous similar message [ 5972.744719] Lustre: lustre-MDT0001: new disk, initializing [ 5972.775103] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5972.787115] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5972.789822] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5974.015011] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5977.472191] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5979.137853] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5981.259669] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5981.353858] Lustre: lustre-OST0000: new disk, initializing [ 5981.355411] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5981.357633] Lustre: 326506:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5981.382310] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5983.122264] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5983.991055] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5983.993957] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5984.027896] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5986.540685] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5993.808368] Lustre: server umount lustre-OST0000 complete [ 5998.560769] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5998.564094] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5998.569650] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5999.585307] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5999.585818] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6004.705392] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6004.707340] Lustre: Skipped 2 previous similar messages [ 6004.731303] Lustre: server umount lustre-MDT0000 complete [ 6005.871046] LustreError: 324532:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783147495 with bad export cookie 8674625530236522420 [ 6005.874243] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6005.874910] LustreError: 324532:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6015.507169] Lustre: server umount lustre-MDT0001 complete [ 6018.770561] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 6019.664122] Key type lgssc unregistered [ 6019.778341] LNet: 328410:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6019.780532] LNetError: 328410:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6019.789228] LNet: Removed LNI 192.168.203.133@tcp [ 6020.067034] Key type .llcrypt unregistered [ 6020.068175] Key type ._llcrypt unregistered [ 6027.244072] Key type ._llcrypt registered [ 6027.245155] Key type .llcrypt registered [ 6027.276769] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 6027.580591] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6027.631371] alg: No test for adler32 (adler32-zlib) [ 6028.473622] Lustre: Lustre: Build Version: 2.17.54_86_gcbe30f7 [ 6028.557531] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 6030.136115] Key type lgssc registered [ 6030.471138] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6032.849227] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6032.855831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6033.989177] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6035.215505] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6037.846308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6037.968496] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6039.267038] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6041.490705] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6043.162567] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6045.264968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6045.369974] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6047.114117] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6049.192090] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6067.712494] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6068.214052] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6069.702749] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.33@tcp (stopping) [ 6070.242594] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6070.243592] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6070.246105] Lustre: Skipped 2 previous similar messages [ 6070.247631] Lustre: Skipped 1 previous similar message [ 6071.574716] Lustre: server umount lustre-MDT0000 complete [ 6072.706043] LustreError: 329641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783147562 with bad export cookie 17758711497540305033 [ 6072.709399] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6072.709614] LustreError: 329641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6074.815799] LustreError: 329655:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6074.820979] LustreError: 329655:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6074.822557] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.33@tcp (stopping) [ 6074.826562] Lustre: Skipped 1 previous similar message [ 6075.360898] LustreError: 330387:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6075.361329] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6077.408771] LustreError: 329655:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6077.408958] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6077.416399] Lustre: Skipped 1 previous similar message [ 6078.884938] Lustre: server umount lustre-MDT0001 complete [ 6085.061771] Lustre: lustre-OST0000: Not available for connect from 192.168.203.33@tcp (stopping) [ 6094.304143] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6094.355199] Lustre: server umount lustre-OST0000 complete [ 6096.944215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6098.157930] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6099.013166] Lustre: 333858:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.133@tcp [ 6099.640906] Lustre: server umount MGS complete [ 6102.211703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6102.358235] LustreError: 334428:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6102.379507] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6103.621108] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6106.131952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6107.438811] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6109.970773] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6110.081450] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6110.083451] Lustre: Skipped 1 previous similar message [ 6111.843495] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6114.011746] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6120.607395] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 6 sec [ 6122.229205] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6125.776305] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 6127.584755] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6127.586850] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6127.590432] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6127.591892] Lustre: Skipped 2 previous similar messages [ 6131.682152] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6131.685668] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6132.876249] Lustre: server umount lustre-OST0000 complete [ 6136.801829] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6140.116693] Lustre: server umount lustre-MDT0000 complete [ 6141.262129] LustreError: 335164:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783147631 with bad export cookie 17758711497540305866 [ 6141.265032] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6141.267416] LustreError: 335164:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6141.376737] Lustre: server umount lustre-MDT0001 complete [ 6144.634307] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 6145.522307] Key type lgssc unregistered [ 6145.645317] LNet: 337864:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6145.647511] LNetError: 337864:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6145.655268] LNet: Removed LNI 192.168.203.133@tcp [ 6145.934182] Key type .llcrypt unregistered [ 6145.935293] Key type ._llcrypt unregistered [ 6151.469896] Key type ._llcrypt registered [ 6151.470903] Key type .llcrypt registered [ 6151.503790] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 6157.239032] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 6157.535508] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6157.572619] alg: No test for adler32 (adler32-zlib) [ 6158.412531] Lustre: Lustre: Build Version: 2.17.54_86_gcbe30f7 [ 6158.492231] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 6160.064135] Key type lgssc registered [ 6160.397735] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6170.861911] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6170.866654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6171.948533] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6171.960618] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6171.995573] Lustre: lustre-MDT0000: new disk, initializing [ 6172.016054] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6172.021983] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6173.256173] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6177.093265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6177.134383] Lustre: 340998:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6177.156159] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6177.157864] Lustre: Skipped 1 previous similar message [ 6177.192230] Lustre: lustre-MDT0001: new disk, initializing [ 6177.222943] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6177.233613] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6177.236651] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6178.463114] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6181.891452] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6183.489924] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6185.542411] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6185.640435] Lustre: lustre-OST0000: new disk, initializing [ 6185.642879] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6185.646415] Lustre: 342961:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6185.674569] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6186.613886] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6186.616843] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6186.650977] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6187.449644] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6190.872356] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6192.169566] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6192.239415] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6193.467117] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6193.529507] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6197.219335] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6197.219381] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6197.220691] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6197.223332] LustreError: Skipped 1 previous similar message [ 6197.226630] Lustre: Skipped 1 previous similar message [ 6200.204967] Lustre: server umount lustre-OST0000 complete [ 6202.337941] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6202.338600] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6202.342426] Lustre: Skipped 1 previous similar message [ 6202.344117] Lustre: Skipped 1 previous similar message [ 6207.457081] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6207.459210] Lustre: Skipped 1 previous similar message [ 6207.514700] Lustre: server umount lustre-MDT0000 complete [ 6208.641123] LustreError: 340990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783147698 with bad export cookie 8673306166923812781 [ 6208.643768] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6208.645091] LustreError: 340990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6208.753472] Lustre: server umount lustre-MDT0001 complete [ 6211.263766] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 02:48:20 (1783147700) [ 6215.984348] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 6218.748535] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 6230.152946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6230.254215] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6230.263132] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6230.303405] Lustre: lustre-MDT0000: new disk, initializing [ 6230.325199] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6230.331401] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6231.857456] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6236.333730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6236.380862] Lustre: 347710:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6236.421661] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6236.426318] Lustre: Skipped 1 previous similar message [ 6236.472204] Lustre: lustre-MDT0001: new disk, initializing [ 6236.510708] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6236.522210] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6236.525886] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6237.843865] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6241.587582] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6243.232416] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6245.347359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6245.464885] Lustre: lustre-OST0000: new disk, initializing [ 6245.466353] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6245.468620] Lustre: 349677:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6245.496776] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6247.246457] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6250.726719] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6255.097263] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6255.102127] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6255.126645] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6265.313558] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6265.316054] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6265.316877] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6265.319705] Lustre: Skipped 1 previous similar message [ 6265.323010] Lustre: Skipped 1 previous similar message [ 6266.701596] Lustre: server umount lustre-OST0000 complete [ 6270.434036] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6270.434631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6270.439318] Lustre: Skipped 1 previous similar message [ 6280.674141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6280.678247] Lustre: Skipped 4 previous similar messages [ 6282.208132] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6282.271964] Lustre: server umount lustre-MDT0000 complete [ 6283.433204] LustreError: 347701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783147773 with bad export cookie 8673306166923813649 [ 6283.436499] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6283.437028] LustreError: 347701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6283.553879] Lustre: server umount lustre-MDT0001 complete [ 6286.920289] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 6287.929791] Key type lgssc unregistered [ 6288.060486] LNet: 351827:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6288.062738] LNetError: 351827:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6288.070384] LNet: Removed LNI 192.168.203.133@tcp [ 6288.378095] Key type .llcrypt unregistered [ 6288.379158] Key type ._llcrypt unregistered [ 6318.191247] Key type ._llcrypt registered [ 6318.192832] Key type .llcrypt registered [ 6318.227598] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 6318.312277] Key type .llcrypt unregistered [ 6318.313389] Key type ._llcrypt unregistered [ 6347.418334] Key type ._llcrypt registered [ 6347.419361] Key type .llcrypt registered [ 6347.450220] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 6347.534325] Key type .llcrypt unregistered [ 6347.535296] Key type ._llcrypt unregistered [ 6358.789692] Key type ._llcrypt registered [ 6358.791287] Key type .llcrypt registered [ 6358.822957] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 6358.912403] Key type .llcrypt unregistered [ 6358.913655] Key type ._llcrypt unregistered [ 6369.681059] Key type ._llcrypt registered [ 6369.682878] Key type .llcrypt registered [ 6369.718955] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 6369.803162] Key type .llcrypt unregistered [ 6369.804256] Key type ._llcrypt unregistered [ 6401.023722] Key type ._llcrypt registered [ 6401.025134] Key type .llcrypt registered [ 6401.076253] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 6401.236568] Key type .llcrypt unregistered [ 6401.238892] Key type ._llcrypt unregistered [ 6414.902261] Key type ._llcrypt registered [ 6414.903713] Key type .llcrypt registered [ 6414.947861] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 6415.072182] Key type .llcrypt unregistered [ 6415.073442] Key type ._llcrypt unregistered [ 6427.931569] Key type ._llcrypt registered [ 6427.932858] Key type .llcrypt registered [ 6427.997332] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 6428.162242] Key type .llcrypt unregistered [ 6428.163689] Key type ._llcrypt unregistered [ 6440.421814] Key type ._llcrypt registered [ 6440.422822] Key type .llcrypt registered [ 6440.463260] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 6440.590232] Key type .llcrypt unregistered [ 6440.591686] Key type ._llcrypt unregistered [ 6450.919959] Key type ._llcrypt registered [ 6450.921286] Key type .llcrypt registered [ 6450.953097] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 6451.040321] Key type .llcrypt unregistered [ 6451.041335] Key type ._llcrypt unregistered [ 6461.882923] Key type ._llcrypt registered [ 6461.883953] Key type .llcrypt registered [ 6461.930653] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 6462.046166] Key type .llcrypt unregistered [ 6462.047215] Key type ._llcrypt unregistered [ 6482.989100] Key type ._llcrypt registered [ 6482.990127] Key type .llcrypt registered [ 6483.023922] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 6483.114169] Key type .llcrypt unregistered [ 6483.115614] Key type ._llcrypt unregistered [ 6493.466851] Key type ._llcrypt registered [ 6493.467956] Key type .llcrypt registered [ 6493.504459] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 6493.616417] Key type .llcrypt unregistered [ 6493.617584] Key type ._llcrypt unregistered [ 6504.729101] Key type ._llcrypt registered [ 6504.730683] Key type .llcrypt registered [ 6504.774203] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 6504.896254] Key type .llcrypt unregistered [ 6504.897935] Key type ._llcrypt unregistered [ 6526.149912] Key type ._llcrypt registered [ 6526.150924] Key type .llcrypt registered [ 6526.192613] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 6533.665529] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 6534.183705] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6534.190711] alg: No test for adler32 (adler32-zlib) [ 6535.040039] Lustre: Lustre: Build Version: 2.17.54_86_gcbe30f7 [ 6535.130642] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 6536.720122] Key type lgssc registered [ 6537.081851] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6547.750595] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6547.755483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6548.829227] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [ 6548.838348] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [ 6548.872109] Lustre: tst250fs-MDT0000: new disk, initializing [ 6548.891191] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6548.897260] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6550.144534] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6554.051334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6554.090931] Lustre: 371749:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6554.112186] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [ 6554.114060] Lustre: Skipped 1 previous similar message [ 6554.148302] Lustre: tst250fs-MDT0001: new disk, initializing [ 6554.176741] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6554.186613] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6554.190492] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6555.385150] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6557.580759] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6559.552684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6559.658661] Lustre: tst250fs-OST0000: new disk, initializing [ 6559.660283] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [ 6559.662490] Lustre: 373385:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6559.688701] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6561.358125] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6565.120845] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6565.173922] Lustre: tst250fs-OST0001: new disk, initializing [ 6565.175450] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [ 6565.177667] Lustre: 374247:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6565.207335] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6565.471389] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6565.474754] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6565.523890] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6565.529565] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6566.922518] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6569.239665] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [ 6579.680760] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6579.682953] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6579.687029] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 6581.217988] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 6581.219115] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6581.221054] Lustre: Skipped 1 previous similar message [ 6581.224428] Lustre: Skipped 2 previous similar messages [ 6584.792349] Lustre: server umount tst250fs-MDT0000 complete [ 6585.855291] LustreError: 371741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783148075 with bad export cookie 12921672295715243249 [ 6585.859037] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6585.860904] LustreError: 371741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6585.968363] Lustre: server umount tst250fs-MDT0001 complete [ 6596.289396] Lustre: server umount tst250fs-OST0000 complete [ 6607.644553] Lustre: server umount tst250fs-OST0001 complete [ 6612.852380] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 6615.058702] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 6624.095496] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 02:55:13 (1783148113) [ 6624.569933] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [ 6629.263923] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 6631.963216] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 6641.053179] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 6311 sec ========== 02:55:30 (1783148130) [ 6641.547978] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 02:55:30 (1783148130) === [ 6642.550853] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 02:55:31 (1783148131) === [ 6654.217014] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 6655.110443] Key type lgssc unregistered [ 6655.225303] LNet: 383356:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6655.227637] LNetError: 383356:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6655.238297] LNet: Removed LNI 192.168.203.133@tcp [ 6655.528095] Key type .llcrypt unregistered [ 6655.529288] Key type ._llcrypt unregistered