[ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 3.10.0-7.9-debug (green@centos7-base) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC) ) #1 SMP Sat Mar 26 23:28:42 EDT 2022 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:centos7:ext4:ro:-p,-b4096 ro crashkernel=128M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000013edfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved [ 0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable [ 0.000000] AGP: No AGP bridge found [ 0.000000] e820: last_pfn = 0x13ee00 max_arch_pfn = 0x400000000 [ 0.000000] MTRR default type: write-back [ 0.000000] MTRR fixed ranges enabled: [ 0.000000] 00000-9FFFF write-back [ 0.000000] A0000-BFFFF uncachable [ 0.000000] C0000-FFFFF write-protect [ 0.000000] MTRR variable ranges enabled: [ 0.000000] 0 base 00C0000000 mask FFC0000000 uncachable [ 0.000000] 1 disabled [ 0.000000] 2 disabled [ 0.000000] 3 disabled [ 0.000000] 4 disabled [ 0.000000] 5 disabled [ 0.000000] 6 disabled [ 0.000000] 7 disabled [ 0.000000] PAT configuration [0-7]: WB WC UC- UC WB WP UC- UC [ 0.000000] e820: last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] mapped at [ffffffffff200b30] [ 0.000000] Base memory trampoline at [ffff880000099000] 99000 size 24576 [ 0.000000] BRK [0x023c1000, 0x023c1fff] PGTABLE [ 0.000000] BRK [0x023c2000, 0x023c2fff] PGTABLE [ 0.000000] BRK [0x023c3000, 0x023c3fff] PGTABLE [ 0.000000] BRK [0x023c4000, 0x023c4fff] PGTABLE [ 0.000000] BRK [0x023c5000, 0x023c5fff] PGTABLE [ 0.000000] BRK [0x023c6000, 0x023c6fff] PGTABLE [ 0.000000] RAMDISK: [mem 0xbc2e2000-0xbffbffff] [ 0.000000] Early table checksum verification disabled [ 0.000000] ACPI: RSDP 00000000000f5950 00014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 00000000bffe1bb7 00034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 00000000bffe1a53 00074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 00000000bffe0040 01A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 00000000bffe0000 00040 [ 0.000000] ACPI: APIC 00000000bffe1ac7 00090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 00000000bffe1b57 00038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 00000000bffe1b8f 00028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x000000013edfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x13e5e3000-0x13e609fff] [ 0.000000] Reserving 128MB of memory at 768MB for crashkernel (System RAM: 4077MB) [ 0.000000] kvm-clock: cpu 0, msr 1:3e592001, primary cpu clock [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 394778665 cycles [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x00001000-0x00ffffff] [ 0.000000] DMA32 [mem 0x01000000-0xffffffff] [ 0.000000] Normal [mem 0x100000000-0x13edfffff] [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x00001000-0x0009efff] [ 0.000000] node 0: [mem 0x00100000-0xbffcdfff] [ 0.000000] node 0: [mem 0x100000000-0x13edfffff] [ 0.000000] Initmem setup node 0 [mem 0x00001000-0x13edfffff] [ 0.000000] On node 0 totalpages: 1043820 [ 0.000000] DMA zone: 64 pages used for memmap [ 0.000000] DMA zone: 21 pages reserved [ 0.000000] DMA zone: 3998 pages, LIFO batch:0 [ 0.000000] DMA32 zone: 12224 pages used for memmap [ 0.000000] DMA32 zone: 782286 pages, LIFO batch:31 [ 0.000000] Normal zone: 4024 pages used for memmap [ 0.000000] Normal zone: 257536 pages, LIFO batch:31 [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] enabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] ACPI: IRQ0 used by override. [ 0.000000] ACPI: IRQ5 used by override. [ 0.000000] ACPI: IRQ9 used by override. [ 0.000000] ACPI: IRQ10 used by override. [ 0.000000] ACPI: IRQ11 used by override. [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] e820: [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 38 pages/cpu s115176 r8192 d32280 u524288 [ 0.000000] pcpu-alloc: s115176 r8192 d32280 u524288 alloc=1*2097152 [ 0.000000] pcpu-alloc: [0] 0 1 2 3 [ 0.000000] KVM setup async PF for cpu 0 [ 0.000000] kvm-stealtime: cpu 0, msr 13e2135c0 [ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 1027487 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:centos7:ext4:ro:-p,-b4096 ro crashkernel=128M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] audit: disabled (until reboot) [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] AGP: Checking aperture... [ 0.000000] AGP: No AGP bridge found [ 0.000000] Memory: 3820268k/5224448k available (8172k kernel code, 1049168k absent, 355012k reserved, 5773k data, 2532k init) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] Hierarchical RCU implementation. [ 0.000000] RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=4. [ 0.000000] Offload RCU callbacks from all CPUs [ 0.000000] Offload RCU callbacks from CPUs: 0-3. [ 0.000000] NR_IRQS:327936 nr_irqs:456 0 [ 0.000000] Console: colour *CGA 80x25 [ 0.000000] console [ttyS1] enabled [ 0.000000] allocated 25165824 bytes of page_cgroup [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ODEBUG: 73 of 73 active objects replaced [ 0.000000] hpet clockevent registered [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.490788] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.493698] pid_max: default: 32768 minimum: 301 [ 0.495342] Security Framework initialized [ 0.496822] SELinux: Initializing. [ 0.498091] SELinux: Starting in permissive mode [ 0.499652] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes) [ 0.503753] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes) [ 0.507223] Mount-cache hash table entries: 8192 (order: 4, 65536 bytes) [ 0.509713] Mountpoint-cache hash table entries: 8192 (order: 4, 65536 bytes) [ 0.513034] Initializing cgroup subsys memory [ 0.514899] Initializing cgroup subsys devices [ 0.516455] Initializing cgroup subsys freezer [ 0.518118] Initializing cgroup subsys net_cls [ 0.519754] Initializing cgroup subsys blkio [ 0.521170] Initializing cgroup subsys perf_event [ 0.522432] Initializing cgroup subsys hugetlb [ 0.523950] Initializing cgroup subsys pids [ 0.525532] Initializing cgroup subsys net_prio [ 0.527904] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.529508] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.531879] tlb_flushall_shift: 6 [ 0.533336] FEATURE SPEC_CTRL Not Present [ 0.536239] FEATURE IBPB_SUPPORT Not Present [ 0.537654] Spectre V2 : Vulnerable [ 0.538755] Speculative Store Bypass: Vulnerable [ 0.540420] debug: unmapping init [mem 0xffffffff82019000-0xffffffff8201ffff] [ 0.549113] ACPI: Core revision 20130517 [ 0.552554] ACPI: All ACPI Tables successfully acquired [ 0.554252] ftrace: allocating 30294 entries in 119 pages [ 0.601510] Enabling x2apic [ 0.602585] Enabled x2apic [ 0.604202] Switched APIC routing to physical x2apic. [ 0.607639] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.609612] smpboot: CPU0: Intel QEMU Virtual CPU version 2.5+ (fam: 0f, model: 6b, stepping: 01) [ 0.713304] Performance Events: unsupported Netburst CPU model 107 no PMU driver, software events only. [ 0.733667] kvm-clock: cpu 1, msr 1:3e592041, secondary cpu clock [ 0.737669] KVM setup async PF for cpu 1 [ 0.738776] kvm-stealtime: cpu 1, msr 13e2935c0 [ 0.751826] kvm-clock: cpu 2, msr 1:3e592081, secondary cpu clock [ 0.756416] KVM setup async PF for cpu 2 [ 0.718545] smpboot: Booting Node 0, Processors #1 #2 #3 OK [ 0.758004] kvm-stealtime: cpu 2, msr 13e3135c0 [ 0.767791] kvm-clock: cpu 3, msr 1:3e5920c1, secondary cpu clock [ 0.775934] Brought up 4 CPUs [ 0.776302] KVM setup async PF for cpu 3 [ 0.776308] kvm-stealtime: cpu 3, msr 13e3935c0 [ 0.779566] smpboot: Max logical packages: 1 [ 0.780793] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.784759] devtmpfs: initialized [ 0.786084] x86/mm: Memory block size: 128MB [ 0.790731] EVM: security.selinux [ 0.791545] EVM: security.ima [ 0.792476] EVM: security.capability [ 0.795492] atomic64 test passed for x86-64 platform with CX8 and with SSE [ 0.797434] NET: Registered protocol family 16 [ 0.798631] cpuidle: using governor haltpoll [ 0.800194] ACPI: bus type PCI registered [ 0.801237] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.803697] PCI: Using configuration type 1 for base access [ 0.811677] ACPI: Added _OSI(Module Device) [ 0.813130] ACPI: Added _OSI(Processor Device) [ 0.814565] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.815385] ACPI: Added _OSI(Processor Aggregator Device) [ 0.816363] ACPI: Added _OSI(Linux-Dell-Video) [ 0.817756] ACPI: EC: Look up EC in DSDT [ 0.820971] ACPI: Interpreter enabled [ 0.822373] ACPI: (supports S0 S3 S4 S5) [ 0.824125] ACPI: Using IOAPIC for interrupt routing [ 0.826083] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.828502] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.834257] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.835438] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI] [ 0.837264] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM [ 0.839521] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.843449] acpiphp: Slot [2] registered [ 0.844617] acpiphp: Slot [3] registered [ 0.846018] acpiphp: Slot [4] registered [ 0.847560] acpiphp: Slot [5] registered [ 0.849050] acpiphp: Slot [6] registered [ 0.850428] acpiphp: Slot [7] registered [ 0.851611] acpiphp: Slot [8] registered [ 0.852646] acpiphp: Slot [9] registered [ 0.854124] acpiphp: Slot [10] registered [ 0.855688] acpiphp: Slot [11] registered [ 0.857165] acpiphp: Slot [12] registered [ 0.858438] acpiphp: Slot [13] registered [ 0.859966] acpiphp: Slot [14] registered [ 0.861494] acpiphp: Slot [15] registered [ 0.862635] acpiphp: Slot [16] registered [ 0.863970] acpiphp: Slot [17] registered [ 0.865118] acpiphp: Slot [18] registered [ 0.866573] acpiphp: Slot [19] registered [ 0.868080] acpiphp: Slot [20] registered [ 0.869339] acpiphp: Slot [21] registered [ 0.870357] acpiphp: Slot [22] registered [ 0.871942] acpiphp: Slot [23] registered [ 0.873328] acpiphp: Slot [24] registered [ 0.874610] acpiphp: Slot [25] registered [ 0.876035] acpiphp: Slot [26] registered [ 0.877488] acpiphp: Slot [27] registered [ 0.878709] acpiphp: Slot [28] registered [ 0.880039] acpiphp: Slot [29] registered [ 0.881406] acpiphp: Slot [30] registered [ 0.882638] acpiphp: Slot [31] registered [ 0.883751] PCI host bridge to bus 0000:00 [ 0.884789] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.886862] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.888969] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.891492] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.894174] pci_bus 0000:00: root bus resource [mem 0x140000000-0x1bfffffff window] [ 0.896853] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.898692] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.899398] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.900165] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.907279] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.910374] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.913806] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.915737] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.917088] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.919311] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.919703] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.921062] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.923326] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.925636] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.938084] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.939999] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.941184] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.945270] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.949686] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.959076] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.969006] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.972711] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.975920] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.988827] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.999992] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 1.006762] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 1.012513] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 1.024477] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 1.031950] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 1.040588] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 1.052752] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 1.081226] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 1.087575] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 1.092860] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 1.097677] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 1.111028] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 1.116760] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 1.121936] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 1.127644] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 1.157707] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 1.165064] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11) [ 1.166990] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) [ 1.169513] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) [ 1.171551] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11) [ 1.173431] ACPI: PCI Interrupt Link [LNKS] (IRQs *9) [ 1.176136] vgaarb: loaded [ 1.176924] SCSI subsystem initialized [ 1.177932] ACPI: bus type USB registered [ 1.179171] usbcore: registered new interface driver usbfs [ 1.180449] usbcore: registered new interface driver hub [ 1.182433] usbcore: registered new device driver usb [ 1.184253] PCI: Using ACPI for IRQ routing [ 1.185105] PCI: pci_cache_line_size set to 64 bytes [ 1.185378] e820: reserve RAM buffer [mem 0x0009fc00-0x0009ffff] [ 1.185383] e820: reserve RAM buffer [mem 0xbffce000-0xbfffffff] [ 1.185385] e820: reserve RAM buffer [mem 0x13ee00000-0x13fffffff] [ 1.185652] NetLabel: Initializing [ 1.186840] NetLabel: domain hash size = 128 [ 1.188233] NetLabel: protocols = UNLABELED CIPSOv4 [ 1.189549] NetLabel: unlabeled traffic allowed by default [ 1.191649] HPET: 3 timers in total, 0 timers will be used for per-cpu timer [ 1.193689] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.195143] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.199667] amd_nb: Cannot enumerate AMD northbridges [ 1.201228] Switched to clocksource kvm-clock [ 1.218478] pnp: PnP ACPI init [ 1.219392] ACPI: bus type PNP registered [ 1.221313] pnp 00:00: Plug and Play ACPI device, IDs PNP0303 (active) [ 1.221406] pnp 00:01: Plug and Play ACPI device, IDs PNP0f13 (active) [ 1.221453] pnp 00:02: [dma 2] [ 1.221493] pnp 00:02: Plug and Play ACPI device, IDs PNP0700 (active) [ 1.221569] pnp 00:03: Plug and Play ACPI device, IDs PNP0501 (active) [ 1.221653] pnp 00:04: Plug and Play ACPI device, IDs PNP0501 (active) [ 1.221720] pnp 00:05: Plug and Play ACPI device, IDs PNP0b00 (active) [ 1.222066] pnp: PnP ACPI: found 6 devices [ 1.223392] ACPI: bus type PNP unregistered [ 1.235985] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.235993] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.235996] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.235998] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.236000] pci_bus 0000:00: resource 8 [mem 0x140000000-0x1bfffffff window] [ 1.236087] NET: Registered protocol family 2 [ 1.237924] TCP established hash table entries: 32768 (order: 6, 262144 bytes) [ 1.240141] TCP bind hash table entries: 32768 (order: 8, 1048576 bytes) [ 1.242923] TCP: Hash tables configured (established 32768 bind 32768) [ 1.245212] TCP: reno registered [ 1.246289] UDP hash table entries: 2048 (order: 5, 196608 bytes) [ 1.248437] UDP-Lite hash table entries: 2048 (order: 5, 196608 bytes) [ 1.250774] NET: Registered protocol family 1 [ 1.252863] RPC: Registered named UNIX socket transport module. [ 1.254941] RPC: Registered udp transport module. [ 1.256501] RPC: Registered tcp transport module. [ 1.257885] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.259423] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.261113] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.262499] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.263923] PCI: CLS 0 bytes, default 64 [ 1.264067] Unpacking initramfs... [ 2.728890] debug: unmapping init [mem 0xffff8800bc2e2000-0xffff8800bffbffff] [ 2.732777] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.734368] software IO TLB [mem 0xb82e2000-0xbc2e2000] (64MB) mapped at [ffff8800b82e2000-ffff8800bc2e1fff] [ 2.739505] cryptomgr_test (52) used greatest stack depth: 14128 bytes left [ 2.740692] futex hash table entries: 1024 (order: 4, 65536 bytes) [ 2.740750] Initialise system trusted keyring [ 2.776967] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 2.789484] zpool: loaded [ 2.790556] zbud: loaded [ 2.791874] VFS: Disk quotas dquot_6.6.0 [ 2.793511] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 2.796404] NFS: Registering the id_resolver key type [ 2.797966] Key type id_resolver registered [ 2.799220] Key type id_legacy registered [ 2.800446] nfs4filelayout_init: NFSv4 File Layout Driver Registering... [ 2.803645] Key type big_key registered [ 2.804900] SELinux: Registering netfilter hooks [ 2.808299] cryptomgr_test (58) used greatest stack depth: 13968 bytes left [ 2.827088] cryptomgr_test (61) used greatest stack depth: 13664 bytes left [ 2.828943] NET: Registered protocol family 38 [ 2.828957] Key type asymmetric registered [ 2.828961] Asymmetric key parser 'x509' registered [ 2.829111] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 250) [ 2.829790] io scheduler noop registered [ 2.829795] io scheduler deadline registered (default) [ 2.829867] io scheduler cfq registered [ 2.829873] io scheduler mq-deadline registered [ 2.829878] io scheduler kyber registered [ 2.832715] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 2.832730] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 [ 2.847513] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.850629] ACPI: Power Button [PWRF] [ 2.854881] GHES: HEST is not enabled! [ 2.903347] ACPI: PCI Interrupt Link [LNKB] enabled at IRQ 10 [ 2.956375] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 11 [ 3.056831] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11 [ 3.108239] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 10 [ 3.223510] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.254213] 00:03: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A [ 3.284099] 00:04: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 3.290631] Non-volatile memory driver v1.3 [ 3.291839] Linux agpgart interface v0.103 [ 3.293005] crash memory driver: version 1.1 [ 3.294590] nbd: registered device at major 43 [ 3.302078] virtio-pci 0000:00:05.0: irq 24 for MSI/MSI-X [ 3.302123] virtio-pci 0000:00:05.0: irq 25 for MSI/MSI-X [ 3.302200] virtio-pci 0000:00:05.0: irq 26 for MSI/MSI-X [ 3.302245] virtio-pci 0000:00:05.0: irq 27 for MSI/MSI-X [ 3.302284] virtio-pci 0000:00:05.0: irq 28 for MSI/MSI-X [ 3.309270] virtio_blk virtio1: [vda] 66816 512-byte logical blocks (34.2 MB/32.6 MiB) [ 3.319157] virtio-pci 0000:00:06.0: irq 29 for MSI/MSI-X [ 3.319218] virtio-pci 0000:00:06.0: irq 30 for MSI/MSI-X [ 3.319254] virtio-pci 0000:00:06.0: irq 31 for MSI/MSI-X [ 3.319297] virtio-pci 0000:00:06.0: irq 32 for MSI/MSI-X [ 3.319345] virtio-pci 0000:00:06.0: irq 33 for MSI/MSI-X [ 3.325745] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.338396] virtio-pci 0000:00:07.0: irq 34 for MSI/MSI-X [ 3.338448] virtio-pci 0000:00:07.0: irq 35 for MSI/MSI-X [ 3.338481] virtio-pci 0000:00:07.0: irq 36 for MSI/MSI-X [ 3.338509] virtio-pci 0000:00:07.0: irq 37 for MSI/MSI-X [ 3.338530] virtio-pci 0000:00:07.0: irq 38 for MSI/MSI-X [ 3.344413] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.350362] virtio-pci 0000:00:08.0: irq 39 for MSI/MSI-X [ 3.350401] virtio-pci 0000:00:08.0: irq 40 for MSI/MSI-X [ 3.350441] virtio-pci 0000:00:08.0: irq 41 for MSI/MSI-X [ 3.350471] virtio-pci 0000:00:08.0: irq 42 for MSI/MSI-X [ 3.350501] virtio-pci 0000:00:08.0: irq 43 for MSI/MSI-X [ 3.357580] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.366040] virtio-pci 0000:00:09.0: irq 44 for MSI/MSI-X [ 3.366090] virtio-pci 0000:00:09.0: irq 45 for MSI/MSI-X [ 3.366141] virtio-pci 0000:00:09.0: irq 46 for MSI/MSI-X [ 3.366195] virtio-pci 0000:00:09.0: irq 47 for MSI/MSI-X [ 3.366237] virtio-pci 0000:00:09.0: irq 48 for MSI/MSI-X [ 3.372742] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.386479] virtio-pci 0000:00:0a.0: irq 49 for MSI/MSI-X [ 3.386534] virtio-pci 0000:00:0a.0: irq 50 for MSI/MSI-X [ 3.386575] virtio-pci 0000:00:0a.0: irq 51 for MSI/MSI-X [ 3.386617] virtio-pci 0000:00:0a.0: irq 52 for MSI/MSI-X [ 3.386659] virtio-pci 0000:00:0a.0: irq 53 for MSI/MSI-X [ 3.393538] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.404561] rdac: device handler registered [ 3.408352] hp_sw: device handler registered [ 3.409774] emc: device handler registered [ 3.411440] libphy: Fixed MDIO Bus: probed [ 3.415987] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 3.419119] ehci-pci: EHCI PCI platform driver [ 3.421173] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 3.423628] ohci-pci: OHCI PCI platform driver [ 3.425636] uhci_hcd: USB Universal Host Controller Interface driver [ 3.428096] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.432369] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.434010] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.437098] mousedev: PS/2 mouse device common for all mice [ 3.440360] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.445811] rtc_cmos 00:05: RTC can wake from S4 [ 3.451038] rtc_cmos 00:05: rtc core: registered rtc_cmos as rtc0 [ 3.454125] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.456938] kvm: host does not support poll control [ 3.458735] kvm: host upgrade recommended [ 3.460942] hidraw: raw HID events driver (C) Jiri Kosina [ 3.462415] usbcore: registered new interface driver usbhid [ 3.464128] usbhid: USB HID core driver [ 3.465419] drop_monitor: Initializing network drop monitor service [ 3.467049] Netfilter messages via NETLINK v0.30. [ 3.468457] TCP: cubic registered [ 3.469210] Initializing XFRM netlink socket [ 3.470905] NET: Registered protocol family 10 [ 3.476290] NET: Registered protocol family 17 [ 3.477934] Key type dns_resolver registered [ 3.480949] mce: Using 10 MCE banks [ 3.482580] Loading compiled-in X.509 certificates [ 3.486902] Loaded X.509 cert 'Magrathea: Glacier signing key: e34d0e1b7fcf5b414cce75d36d8482945c781ed6' [ 3.489974] registered taskstats version 1 [ 3.493964] modprobe (72) used greatest stack depth: 13456 bytes left [ 3.502776] Key type trusted registered [ 3.513670] Key type encrypted registered [ 3.515259] IMA: No TPM chip found, activating TPM-bypass! (rc=-19) [ 3.518795] BERT: Boot Error Record Table support is disabled. Enable it by using bert_enable as kernel parameter. [ 3.523806] rtc_cmos 00:05: setting system clock to 2024-03-07 04:14:42 UTC (1709784882) [ 3.527655] debug: unmapping init [mem 0xffffffff81da0000-0xffffffff82018fff] [ 3.529598] Write protecting the kernel read-only data: 12288k [ 3.532580] debug: unmapping init [mem 0xffff8800017fe000-0xffff8800017fffff] [ 3.535534] debug: unmapping init [mem 0xffff880001b9b000-0xffff880001bfffff] [ 3.551642] random: systemd: uninitialized urandom read (16 bytes read) [ 3.554480] random: systemd: uninitialized urandom read (16 bytes read) [ 3.556656] random: systemd: uninitialized urandom read (16 bytes read) [ 3.561268] systemd[1]: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN) [ 3.565823] systemd[1]: Detected virtualization kvm. [ 3.567016] systemd[1]: Detected architecture x86-64. [ 3.568234] systemd[1]: Running in initial RAM disk. [ 3.572461] systemd[1]: No hostname configured. [ 3.573716] systemd[1]: Set hostname to . [ 3.576239] random: systemd: uninitialized urandom read (16 bytes read) [ 3.580076] systemd[1]: Initializing machine ID from random generator. [ 3.630408] dracut-rootfs-g (86) used greatest stack depth: 13264 bytes left [ 3.633231] random: systemd: uninitialized urandom read (16 bytes read) [ 3.635077] random: systemd: uninitialized urandom read (16 bytes read) [ 3.637094] random: systemd: uninitialized urandom read (16 bytes read) [ 3.639036] random: systemd: uninitialized urandom read (16 bytes read) [ 3.642947] random: systemd: uninitialized urandom read (16 bytes read) [ 3.646220] random: systemd: uninitialized urandom read (16 bytes read) [ 3.656034] systemd[1]: Reached target Timers. [ 3.659702] systemd[1]: Reached target Local File Systems. [ 3.664383] systemd[1]: Reached target Swap. [ 3.668244] systemd[1]: Created slice Root Slice. [ 3.672589] systemd[1]: Listening on Journal Socket. [ 3.676705] systemd[1]: Created slice System Slice. [ 3.684378] systemd[1]: Starting Journal Service... [ 3.689553] systemd[1]: Starting Load Kernel Modules... [ 3.697483] systemd[1]: Starting Setup Virtual Console... [ 3.701517] systemd[1]: Reached target Slices. [ 3.708411] systemd[1]: Starting Create list of required static device nodes for the current kernel... [ 3.722887] systemd[1]: Starting dracut cmdline hook... [ 3.726813] systemd[1]: Listening on udev Kernel Socket. [ 3.730582] systemd[1]: Listening on udev Control Socket. [ 3.734630] systemd[1]: Reached target Sockets. [ 3.740491] tsc: Refined TSC clocksource calibration: 2399.949 MHz [ 3.757680] systemd[1]: Started Journal Service. [ 3.851014] random: fast init done [ 4.287224] input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input2 [ 4.318461] libata version 3.00 loaded. [ 4.327361] ata_piix 0000:00:01.1: version 2.13 [ 4.343586] scsi host0: ata_piix [ 4.353749] scsi host1: ata_piix [ 4.355042] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.357750] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 4.440406] ip (319) used greatest stack depth: 13080 bytes left [ 4.491437] ip (342) used greatest stack depth: 12360 bytes left [ 7.574237] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ 8.003439] systemd-journald[100]: Received SIGTERM from PID 1 (systemd). [ 8.238338] SELinux: Disabled at runtime. [ 8.239896] SELinux: Unregistering netfilter hooks [ 8.303076] ip_tables: (C) 2000-2006 Netfilter Core Team [ 8.305935] systemd[1]: Inserted module 'ip_tables' [ 8.726166] systemd-tmpfile (578) used greatest stack depth: 11824 bytes left [ 8.766494] systemd-journald[565]: Received request to flush runtime journal from PID 1 [ 8.932454] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 8.959158] input: PC Speaker as /devices/platform/pcspkr/input/input3 [ 9.020533] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ 9.023793] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ 12.118757] mount.nfs (732) used greatest stack depth: 10728 bytes left [ 25.664216] libcfs: loading out-of-tree module taints kernel. [ 25.666159] libcfs: module verification failed: signature and/or required key missing - tainting kernel [ 31.009329] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 31.024913] alg: No test for adler32 (adler32-zlib) [ 31.281093] PCLMULQDQ-NI instructions are not detected. [ 31.918380] Lustre: Lustre: Build Version: 2.15.61_117_gc7b1037 [ 32.099132] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 32.100737] LNet: Accept secure, port 988 [ 33.644619] Key type lgssc registered [ 33.975016] Lustre: Echo OBD driver; http://www.lustre.org/ [ 34.433703] icp: module license 'CDDL' taints kernel. [ 34.435815] Disabling lock debugging due to kernel taint [ 36.872428] ZFS: Loaded module v0.8.6-1, ZFS pool version 5000, ZFS filesystem version 5 [ 39.160065] vdc: vdc1 vdc9 [ 41.678521] vde: vde1 vde9 [ 43.971661] vdf: vdf1 vdf9 [ 47.248670] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 48.854248] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 49.965692] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 50.033056] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 50.068991] Lustre: lustre-MDT0000: new disk, initializing [ 50.149149] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 50.166034] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 50.224956] mount.lustre (6243) used greatest stack depth: 10104 bytes left [ 50.942531] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 53.425286] random: crng init done [ 54.232724] Lustre: lustre-OST0000: new disk, initializing [ 54.234976] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 54.236788] Lustre: Skipped 1 previous similar message [ 54.250896] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 55.370255] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 55.372819] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 55.406667] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 55.599993] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 58.706677] Lustre: lustre-OST0001: new disk, initializing [ 58.708558] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 58.726084] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 60.109450] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 63.480168] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 63.483275] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 63.519096] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 64.550907] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 67.752788] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 73.396982] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing check_logdir /tmp/testlogs/ [ 74.289531] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing yml_node [ 75.311667] Lustre: DEBUG MARKER: Client: 2.15.61.117 [ 75.895646] Lustre: DEBUG MARKER: MDS: 2.15.61.117 [ 76.505826] Lustre: DEBUG MARKER: OSS: 2.15.61.117 [ 76.887969] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Mar 6 23:15:55 EST 2024 [ 78.364356] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball 110 [ 78.732645] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 93.487560] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 93.487680] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 93.497896] Lustre: Skipped 1 previous similar message [ 95.550621] Lustre: server umount lustre-MDT0000 complete [ 96.511785] LustreError: 6318:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709784975 with bad export cookie 10190371261446279947 [ 96.512552] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 96.518489] LustreError: 6318:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 102.518943] Lustre: server umount lustre-OST0000 complete [ 105.510357] Lustre: server umount lustre-OST0001 complete [ 106.946927] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_hostid [ 108.283309] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 110.221882] vdc: vdc1 vdc9 [ 112.380137] vde: vde1 vde9 [ 114.623552] vdf: vdf1 vdf9 [ 116.704423] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 116.724475] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 116.752887] Lustre: lustre-MDT0000: new disk, initializing [ 116.807053] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 116.818946] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 117.556246] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 119.744345] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 121.116637] Lustre: lustre-OST0000: new disk, initializing [ 121.118088] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 121.120041] Lustre: Skipped 1 previous similar message [ 121.138701] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 121.175496] mount.lustre (18173) used greatest stack depth: 9544 bytes left [ 122.323067] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 122.353802] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 122.356422] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 122.388697] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 124.568116] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 125.311927] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 125.358201] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 127.344082] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 127.347867] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 127.348219] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 127.348221] Lustre: Skipped 1 previous similar message [ 127.868512] Lustre: server umount lustre-OST0000 complete [ 128.946511] Lustre: server umount lustre-MDT0000 complete [ 131.080729] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 23:16:50 (1709785010) [ 132.295349] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 132.966567] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 133.912477] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 136.352054] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 137.405869] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 144.020208] Lustre: Modifying parameter general.osc.*.max_dirty_mb in log params [ 147.151851] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 147.155738] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 148.847468] Lustre: server umount lustre-MDT0000 complete [ 149.908064] LustreError: 21098:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709785028 with bad export cookie 10190371261446281333 [ 149.909058] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 149.913848] LustreError: 21098:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 151.953887] Lustre: server umount lustre-OST0000 complete [ 155.805492] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 157.598054] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 157.600228] Lustre: Skipped 1 previous similar message [ 158.373734] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 161.618015] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 163.599823] Lustre: lustre-OST0001: new disk, initializing [ 163.601325] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 164.908315] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 165.274072] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 165.277234] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 165.302851] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 168.738856] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 169.615983] Lustre: Setting parameter general.lod.*.mdt_hash in log params [ 177.538970] Lustre: server umount lustre-MDT0000 complete [ 178.557867] LustreError: 24869:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709785057 with bad export cookie 10190371261446281851 [ 178.562424] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 188.557313] Lustre: 29283:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1709785057/real 1709785057] req@ffff88008f4eb800 x1792839422878336/t0(0) o39->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1709785067 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 [ 188.576944] Lustre: server umount lustre-OST0000 complete [ 199.614401] Lustre: 29631:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1709785068/real 1709785068] req@ffff88012b42a300 x1792839422878784/t0(0) o39->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1709785078 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 [ 199.669005] Lustre: server umount lustre-OST0001 complete [ 202.691502] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 204.523100] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 204.524822] Lustre: Skipped 2 previous similar messages [ 205.302244] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 208.134614] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 210.843542] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 218.315222] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 219.141509] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 219.143832] Lustre: Skipped 1 previous similar message [ 227.376813] Lustre: server umount lustre-MDT0000 complete [ 228.400029] LustreError: 1734:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709785107 with bad export cookie 10190371261446282656 [ 228.402988] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 238.399542] Lustre: 2226:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1709785107/real 1709785107] req@ffff88008f4e9880 x1792839422887744/t0(0) o39->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1709785117 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 [ 238.414893] Lustre: server umount lustre-OST0000 complete [ 249.387526] Lustre: 2555:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1709785118/real 1709785118] req@ffff88008f4e9880 x1792839422888192/t0(0) o39->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1709785128 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 [ 249.446937] Lustre: server umount lustre-OST0001 complete [ 253.070299] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 23:18:52 (1709785132) [ 258.056667] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 260.604783] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 263.593842] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 266.615525] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 274.241296] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 275.131773] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 283.151217] Lustre: server umount lustre-MDT0000 complete [ 284.177028] LustreError: 5119:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709785163 with bad export cookie 10190371261446283335 [ 284.181961] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 294.176747] Lustre: 8845:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1709785163/real 1709785163] req@ffff88008f4e9880 x1792839422897152/t0(0) o39->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1709785173 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 [ 294.194681] Lustre: server umount lustre-OST0000 complete [ 305.193785] Lustre: 9181:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1709785174/real 1709785174] req@ffff88008f4e9880 x1792839422897600/t0(0) o39->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1709785184 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 [ 305.255363] Lustre: server umount lustre-OST0001 complete [ 309.101946] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 23:19:48 (1709785188) [ 311.958498] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 313.795723] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 313.797378] Lustre: Skipped 5 previous similar messages [ 314.537535] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 317.298934] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 320.054974] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 327.525602] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 328.329094] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 335.623999] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.11@tcp (stopping) [ 336.380896] Lustre: server umount lustre-MDT0000 complete [ 338.094682] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 338.185742] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 338.959685] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 339.988263] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 341.374313] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 1 sec [ 343.215733] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 343.217248] Lustre: Skipped 1 previous similar message [ 348.366733] Lustre: server umount lustre-MDT0000 complete [ 349.410921] LustreError: 13157:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709785228 with bad export cookie 10190371261446284714 [ 349.414088] LustreError: 13157:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 349.421316] Lustre: server umount lustre-OST0000 complete [ 350.497396] Lustre: server umount lustre-OST0001 complete [ 354.063323] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 23:20:33 (1709785233) [ 356.720706] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 359.280738] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 362.337620] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 365.341390] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 372.899573] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 383.136038] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 383.136452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 383.136453] Lustre: Skipped 2 previous similar messages [ 383.144142] Lustre: Skipped 2 previous similar messages [ 388.473851] Lustre: server umount lustre-MDT0000 complete [ 389.513284] LustreError: 18841:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709785268 with bad export cookie 10190371261446285148 [ 389.514710] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 389.514712] LustreError: Skipped 1 previous similar message [ 389.523964] LustreError: 18841:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 389.530132] Lustre: server umount lustre-OST0000 complete [ 390.600628] Lustre: server umount lustre-OST0001 complete [ 394.175720] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 23:21:13 (1709785273) [ 397.950419] Lustre: Setting parameter test1234-MDT0000.mdt.identity_upcall in log test1234-MDT0000 [ 397.952364] Lustre: Skipped 3 previous similar messages [ 397.979502] Lustre: ctl-test1234-MDT0000: No data found on store. Initialize space: rc = -61 [ 398.016214] Lustre: test1234-MDT0000: new disk, initializing [ 398.113629] Lustre: ctl-test1234-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 398.996783] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 405.921544] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 406.118287] Lustre: cli-test1234-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 406.141423] Lustre: test1234-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 411.104337] Lustre: test1234-MDT0000-lwp-OST0000: Connection to test1234-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 411.108852] Lustre: test1234-MDT0000: Not available for connect from 0@lo (stopping) [ 411.110224] Lustre: Skipped 3 previous similar messages [ 414.468961] Lustre: server umount test1234-MDT0000 complete [ 415.601211] Lustre: server umount test1234-OST0000 complete [ 419.431082] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 23:21:38 (1709785298) [ 419.827469] Lustre: DEBUG MARKER: SKIP: conf-sanity test_78 ldiskfs only test [ 421.850308] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 23:21:40 (1709785300) [ 423.338501] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 426.237003] vdc: vdc1 vdc9 [ 432.482394] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_hostid [ 433.961465] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 436.014869] vdc: vdc1 vdc9 [ 438.469152] vde: vde1 vde9 [ 441.014583] vdf: vdf1 vdf9 [ 443.376613] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 443.379846] Lustre: Skipped 2 previous similar messages [ 443.411863] Lustre: lustre-MDT0000: new disk, initializing [ 443.413143] Lustre: Skipped 1 previous similar message [ 443.478107] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 443.480268] Lustre: Skipped 8 previous similar messages [ 443.495408] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 443.497814] Lustre: Skipped 1 previous similar message [ 444.306785] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 446.622479] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 449.290815] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 449.979270] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 450.004487] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 451.558282] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 452.392162] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 452.439358] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 454.976147] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 454.976309] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 454.982132] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 458.975703] Lustre: server umount lustre-OST0000 complete [ 460.161874] Lustre: server umount lustre-MDT0000 complete [ 463.814937] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 23:22:22 (1709785342) [ 465.938115] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 466.991808] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 469.550521] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 470.615290] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 471.565652] Lustre: 12808:0:(genops.c:1641:obd_export_evict_by_uuid()) MGS: evicting 4d1cf4c2-4f59-4e48-9ab1-6cbdd38c1765 at adminstrative request [ 473.327755] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 473.330395] LustreError: Skipped 1 previous similar message [ 473.331978] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 483.332117] Lustre: MGS: Client 4d1cf4c2-4f59-4e48-9ab1-6cbdd38c1765 (at 0@lo) reconnecting [ 483.334313] Lustre: Evicted from MGS (at 192.168.204.111@tcp) after server handle changed from 0x8d6b78abdfdaef43 to 0x8d6b78abdfdaf0a8 [ 483.337808] Lustre: MGC192.168.204.111@tcp: Connection restored to 192.168.204.111@tcp (at 0@lo) [ 504.446572] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 504.877911] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 504.901818] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 506.835577] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 509.856601] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 513.634055] Lustre: server umount lustre-OST0001 complete [ 514.863851] LustreError: 137-5: lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 519.872680] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 519.872746] LustreError: 137-5: lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 519.879352] Lustre: Skipped 2 previous similar messages [ 520.813372] Lustre: server umount lustre-OST0000 complete [ 521.993514] Lustre: server umount lustre-MDT0000 complete [ 525.643518] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 23:23:24 (1709785404) [ 525.997004] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 528.020366] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 23:23:27 (1709785407) [ 528.408136] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 530.500003] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 23:23:29 (1709785409) [ 530.910331] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 533.073105] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 23:23:32 (1709785412) [ 533.484552] Lustre: DEBUG MARKER: SKIP: conf-sanity test_83 ldiskfs only test [ 535.427757] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 23:23:34 (1709785414) [ 537.428558] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 538.401083] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 540.774786] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 541.869963] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 542.703523] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 546.574676] LustreError: 137-5: lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 546.812556] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 549.512056] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 550.611723] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 551.424128] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 551.473249] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 558.178512] LustreError: 22662:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 558.422422] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 563.441585] Lustre: Failing over lustre-MDT0000 [ 563.597062] Lustre: server umount lustre-MDT0000 complete [ 575.286373] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 575.374708] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 575.378168] Lustre: Skipped 2 previous similar messages [ 576.182524] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 576.698126] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 576.698790] Lustre: 23977:0:(ldlm_lib.c:1992:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 576.728722] LustreError: 24071:0:(ldlm_lib.c:2625:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 577.031407] LustreError: 24071:0:(ldlm_lib.c:2625:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 577.436006] Lustre: 24071:0:(ldlm_lib.c:1992:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 577.439803] Lustre: 24071:0:(ldlm_lib.c:1992:extend_recovery_timer()) Skipped 7 previous similar messages [ 577.442022] LustreError: 24071:0:(ldlm_lib.c:2625:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 577.445209] LustreError: 24071:0:(ldlm_lib.c:2625:replay_request_or_update()) Skipped 1 previous similar message [ 577.748409] LustreError: 24071:0:(ldlm_lib.c:2625:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 577.751092] LustreError: 24071:0:(ldlm_lib.c:2625:replay_request_or_update()) Skipped 1 previous similar message [ 578.554835] Lustre: 24071:0:(ldlm_lib.c:1992:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 578.558344] Lustre: 24071:0:(ldlm_lib.c:1992:extend_recovery_timer()) Skipped 2 previous similar messages [ 578.560240] LustreError: 24071:0:(ldlm_lib.c:2625:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 578.563110] LustreError: 24071:0:(ldlm_lib.c:2625:replay_request_or_update()) Skipped 2 previous similar messages [ 578.865441] LustreError: 24071:0:(ldlm_lib.c:2625:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 578.867756] LustreError: 24071:0:(ldlm_lib.c:2625:replay_request_or_update()) Skipped 2 previous similar messages [ 580.401114] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 192.168.204.111@tcp (at 0@lo) [ 580.579340] Lustre: 24071:0:(ldlm_lib.c:1992:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 580.583604] Lustre: 24071:0:(ldlm_lib.c:1992:extend_recovery_timer()) Skipped 5 previous similar messages [ 580.587353] LustreError: 24071:0:(ldlm_lib.c:2625:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 580.592061] LustreError: 24071:0:(ldlm_lib.c:2625:replay_request_or_update()) Skipped 5 previous similar messages [ 580.895422] LustreError: 24071:0:(ldlm_lib.c:2625:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 580.898266] LustreError: 24071:0:(ldlm_lib.c:2625:replay_request_or_update()) Skipped 5 previous similar messages [ 584.400430] Lustre: 2606:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1709785447/real 1709785447] req@ffff88009f2df480 x1792839422959936/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1709785463 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 584.406002] Lustre: 2606:0:(client.c:2338:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 584.810383] Lustre: 24071:0:(ldlm_lib.c:1992:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 584.814079] Lustre: 24071:0:(ldlm_lib.c:1992:extend_recovery_timer()) Skipped 10 previous similar messages [ 584.817212] LustreError: 24071:0:(ldlm_lib.c:2625:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 584.819263] LustreError: 24071:0:(ldlm_lib.c:2625:replay_request_or_update()) Skipped 10 previous similar messages [ 585.120326] LustreError: 24071:0:(ldlm_lib.c:2625:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 585.123604] LustreError: 24071:0:(ldlm_lib.c:2625:replay_request_or_update()) Skipped 10 previous similar messages [ 592.852709] Lustre: 24071:0:(ldlm_lib.c:1992:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 592.856559] Lustre: 24071:0:(ldlm_lib.c:1992:extend_recovery_timer()) Skipped 21 previous similar messages [ 592.858944] LustreError: 24071:0:(ldlm_lib.c:2625:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 592.861772] LustreError: 24071:0:(ldlm_lib.c:2625:replay_request_or_update()) Skipped 21 previous similar messages [ 593.164360] LustreError: 24071:0:(ldlm_lib.c:2625:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 593.166425] LustreError: 24071:0:(ldlm_lib.c:2625:replay_request_or_update()) Skipped 21 previous similar messages [ 609.211406] LustreError: 24071:0:(ldlm_lib.c:2625:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 609.213889] LustreError: 24071:0:(ldlm_lib.c:2625:replay_request_or_update()) Skipped 42 previous similar messages [ 609.218086] Lustre: 24071:0:(ldlm_lib.c:1992:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 609.221747] Lustre: 24071:0:(ldlm_lib.c:1992:extend_recovery_timer()) Skipped 43 previous similar messages [ 609.224679] LustreError: 24071:0:(ldlm_lib.c:2625:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 609.227942] LustreError: 24071:0:(ldlm_lib.c:2625:replay_request_or_update()) Skipped 43 previous similar messages [ 636.698367] Lustre: 24071:0:(ldlm_lib.c:2300:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 636.702712] Lustre: 24071:0:(ldlm_lib.c:2310:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 636.706686] Lustre: 24071:0:(genops.c:1498:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 6a1e4daa-2e6d-4dc1-9176-345f2d1b9ba8@192.168.204.11@tcp [ 636.710407] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 636.712449] Lustre: 24071:0:(ldlm_lib.c:1824:abort_req_replay_queue()) @@@ aborted: req@ffff880098097480 x1792839417653184/t0(12884902058) o35->6a1e4daa-2e6d-4dc1-9176-345f2d1b9ba8@192.168.204.11@tcp:452/0 lens 392/0 e 0 to 0 dl 1709785532 ref 1 fl Complete:/204/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 [ 636.725892] Lustre: lustre-MDT0000: Denying connection for new client 6a1e4daa-2e6d-4dc1-9176-345f2d1b9ba8 (at 192.168.204.11@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 1 evicted) already passed deadline 0:00 [ 636.734777] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 636.753275] Lustre: lustre-MDT0000: Recovery over after 1:00, of 2 clients 1 recovered and 1 was evicted. [ 636.769494] Lustre: lustre-OST0000: deleting orphan objects from 0x240000400:108 to 0x240000400:609 [ 636.769636] Lustre: lustre-OST0001: deleting orphan objects from 0x280000400:75 to 0x280000400:577 [ 649.916556] Lustre: server umount lustre-OST0000 complete [ 650.511903] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 651.952061] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 651.954689] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 651.956305] Lustre: Skipped 1 previous similar message [ 653.012613] Lustre: server umount lustre-OST0001 complete [ 654.142540] Lustre: server umount lustre-MDT0000 complete [ 657.813532] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 23:25:36 (1709785536) [ 676.404725] LustreError: 27378:0:(mgc_request.c:251:do_config_log_add()) MGC192.168.204.111@tcp: failed processing log, type 1: rc = -5 [ 692.410866] LustreError: 27378:0:(mgc_request.c:251:do_config_log_add()) MGC192.168.204.111@tcp: failed processing log, type 4: rc = -110 [ 841.431410] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 841.433541] Lustre: Skipped 8 previous similar messages [ 842.795778] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 843.831006] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 844.373316] Lustre: server umount lustre-OST0000 complete [ 847.928898] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 23:28:47 (1709785727) [ 848.275084] Lustre: DEBUG MARKER: SKIP: conf-sanity test_86 LU-6442: no such mkfs params for ZFS OSTs [ 850.126750] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 23:28:49 (1709785729) [ 850.441259] Lustre: DEBUG MARKER: SKIP: conf-sanity test_87 ldiskfs only test [ 852.218838] Lustre: DEBUG MARKER: == conf-sanity test 88: check the default mount options can be overridden ========================================================== 23:28:51 (1709785731) [ 852.559961] Lustre: DEBUG MARKER: SKIP: conf-sanity test_88 LU-6662: no implementation for ZFS [ 854.522432] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 23:28:53 (1709785733) [ 867.433494] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 869.722637] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 871.567501] Lustre: server umount lustre-MDT0000 complete [ 880.434729] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_hostid [ 882.004912] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 884.169239] vdc: vdc1 vdc9 [ 886.405861] vde: vde1 vde9 [ 888.765653] vdf: vdf1 vdf9 [ 891.017323] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 891.020821] Lustre: Skipped 1 previous similar message [ 891.041121] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 891.043018] Lustre: Skipped 3 previous similar messages [ 891.072766] Lustre: lustre-MDT0000: new disk, initializing [ 891.073944] Lustre: Skipped 2 previous similar messages [ 891.149381] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 891.154915] Lustre: Skipped 2 previous similar messages [ 891.956746] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 894.283120] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 896.925174] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 896.954057] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 897.067072] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 899.408610] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 900.207829] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 900.255113] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 901.919968] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 901.920204] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 901.920206] Lustre: Skipped 2 previous similar messages [ 902.811744] Lustre: server umount lustre-OST0000 complete [ 903.989108] Lustre: server umount lustre-MDT0000 complete [ 907.699891] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 23:29:46 (1709785786) [ 909.849170] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 910.902857] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 913.442456] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 914.518332] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 921.279616] Lustre: *** cfs_fail_loc=159, val=0*** [ 937.283163] Lustre: lustre-MDT0000: Client c5fbe080-5717-49a2-b60a-cdad6b74f6ad (at 192.168.204.11@tcp) reconnecting [ 937.546762] Lustre: *** cfs_fail_loc=159, val=0*** [ 937.548037] Lustre: Skipped 5 previous similar messages [ 953.552854] Lustre: lustre-MDT0000: Client c5fbe080-5717-49a2-b60a-cdad6b74f6ad (at 192.168.204.11@tcp) reconnecting [ 954.271986] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 954.272228] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 954.272230] Lustre: Skipped 1 previous similar message [ 956.269383] Lustre: server umount lustre-OST0000 complete [ 957.434525] Lustre: server umount lustre-MDT0000 complete [ 960.407638] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 960.877388] Key type lgssc unregistered [ 960.939641] LNet: 13988:0:(lib-ptl.c:990:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 960.943267] LNet: Removed LNI 192.168.204.111@tcp [ 969.106176] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 969.125991] alg: No test for adler32 (adler32-zlib) [ 969.938770] Lustre: Lustre: Build Version: 2.15.61_117_gc7b1037 [ 969.982327] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 969.984056] LNet: Accept secure, port 988 [ 971.509325] Key type lgssc registered [ 971.748699] Lustre: Echo OBD driver; http://www.lustre.org/ [ 973.125527] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 974.330803] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 975.069266] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 976.096887] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 977.404626] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 978.621649] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 979.637641] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 984.401759] Lustre: lustre-OST0000: deleting orphan objects from 0x240000400:42 to 0x240000400:65 [ 989.304299] Lustre: *** cfs_fail_loc=159, val=0*** [ 1005.315460] Lustre: lustre-MDT0000: Client 53af0de5-c06d-4c03-b40d-580d60d77047 (at 192.168.204.11@tcp) reconnecting [ 1005.706685] Lustre: *** cfs_fail_loc=159, val=0*** [ 1021.710376] Lustre: lustre-MDT0000: Client 53af0de5-c06d-4c03-b40d-580d60d77047 (at 192.168.204.11@tcp) reconnecting [ 1021.979396] Lustre: *** cfs_fail_loc=159, val=0*** [ 1021.981579] Lustre: Skipped 4 previous similar messages [ 1037.983508] Lustre: lustre-MDT0000: Client 53af0de5-c06d-4c03-b40d-580d60d77047 (at 192.168.204.11@tcp) reconnecting [ 1038.714072] Lustre: 18307:0:(mdt_handler.c:84:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 1039.426297] Lustre: *** cfs_fail_loc=159, val=0*** [ 1039.428694] Lustre: Skipped 2 previous similar messages [ 1055.429346] Lustre: lustre-MDT0000: Client 3dc5a602-4eeb-443c-ba01-95398adf5268 (at 192.168.204.11@tcp) reconnecting [ 1055.717336] Lustre: *** cfs_fail_loc=159, val=0*** [ 1055.719258] Lustre: Skipped 13 previous similar messages [ 1071.722025] Lustre: lustre-MDT0000: Client 3dc5a602-4eeb-443c-ba01-95398adf5268 (at 192.168.204.11@tcp) reconnecting [ 1071.957011] Lustre: 18834:0:(mdt_handler.c:84:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 1074.544069] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1074.544142] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1074.544535] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1074.846725] Lustre: server umount lustre-OST0000 complete [ 1075.984599] Lustre: server umount lustre-MDT0000 complete [ 1078.899977] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 1079.369708] Key type lgssc unregistered [ 1079.432746] LNet: 19877:0:(lib-ptl.c:990:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1079.435633] LNet: Removed LNI 192.168.204.111@tcp [ 1086.909668] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1086.951670] alg: No test for adler32 (adler32-zlib) [ 1087.765380] Lustre: Lustre: Build Version: 2.15.61_117_gc7b1037 [ 1087.812444] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 1087.814705] LNet: Accept secure, port 988 [ 1089.341612] Key type lgssc registered [ 1089.521083] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1090.754077] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1091.936665] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1092.749285] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1093.812541] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1095.161889] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1096.553168] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1097.664112] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1101.158173] Lustre: lustre-OST0000: deleting orphan objects from 0x240000400:91 to 0x240000400:129 [ 1105.003661] Lustre: 23534:0:(mdt_handler.c:84:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 1105.736245] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight in log params [ 1106.539547] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight in log params [ 1106.541440] Lustre: Skipped 1 previous similar message [ 1107.053558] Lustre: 24000:0:(mdt_handler.c:84:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 1107.057455] Lustre: 24000:0:(mdt_handler.c:84:mdt_max_mod_rpcs_per_client_set()) Skipped 1 previous similar message [ 1111.168247] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1111.168373] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1111.168651] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1113.793547] Lustre: server umount lustre-OST0000 complete [ 1114.965368] Lustre: server umount lustre-MDT0000 complete [ 1118.036902] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 1118.529596] Key type lgssc unregistered [ 1118.601695] LNet: 25070:0:(lib-ptl.c:990:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1118.604577] LNet: Removed LNI 192.168.204.111@tcp [ 1126.545813] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1126.564898] alg: No test for adler32 (adler32-zlib) [ 1127.374603] Lustre: Lustre: Build Version: 2.15.61_117_gc7b1037 [ 1127.417450] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 1127.419376] LNet: Accept secure, port 988 [ 1128.948581] Key type lgssc registered [ 1129.148133] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1130.505092] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1131.700746] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1132.483367] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1133.627407] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1135.051518] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1136.381800] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1137.551055] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1141.047346] Lustre: lustre-OST0000: deleting orphan objects from 0x240000400:91 to 0x240000400:161 [ 1144.855763] Lustre: *** cfs_fail_loc=159, val=0*** [ 1160.859460] Lustre: lustre-MDT0000: Client 566a429d-dc9f-4036-ba6a-7bbd62a8d43a (at 192.168.204.11@tcp) reconnecting [ 1162.664532] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 1162.666871] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1162.671766] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1166.079746] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1167.674694] Lustre: server umount lustre-OST0000 complete [ 1168.951314] Lustre: server umount lustre-MDT0000 complete [ 1172.082892] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 1172.609780] Key type lgssc unregistered [ 1172.681650] LNet: 30013:0:(lib-ptl.c:990:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1172.683909] LNet: Removed LNI 192.168.204.111@tcp [ 1180.983320] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1181.003589] alg: No test for adler32 (adler32-zlib) [ 1181.820685] Lustre: Lustre: Build Version: 2.15.61_117_gc7b1037 [ 1181.866857] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 1181.868486] LNet: Accept secure, port 988 [ 1183.396457] Key type lgssc registered [ 1183.615150] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1184.980392] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1186.171008] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1187.000237] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1188.078765] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1189.503420] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1190.843318] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1191.980590] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1196.497871] Lustre: lustre-OST0000: deleting orphan objects from 0x240000400:170 to 0x240000400:193 [ 1209.225843] Lustre: lustre-OST0000: evicting 12926704-a326-4800-99f4-c945f63fa429 (at 192.168.204.11@tcp) by administrative request [ 1236.559925] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1236.559927] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1236.567231] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1238.792751] Lustre: server umount lustre-OST0000 complete [ 1240.006648] Lustre: server umount lustre-MDT0000 complete [ 1243.111924] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 1243.610053] Key type lgssc unregistered [ 1243.677711] LNet: 3097:0:(lib-ptl.c:990:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1243.683242] LNet: Removed LNI 192.168.204.111@tcp [ 1272.824085] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1272.862441] alg: No test for adler32 (adler32-zlib) [ 1273.675918] Lustre: Lustre: Build Version: 2.15.61_117_gc7b1037 [ 1273.731123] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 1273.733005] LNet: Accept secure, port 988 [ 1275.263693] Key type lgssc registered [ 1275.503352] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1276.904288] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1278.127995] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1279.000891] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1280.174425] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1281.582719] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1283.017737] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1284.220798] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1288.577780] Lustre: lustre-OST0000: deleting orphan objects from 0x240000400:195 to 0x240000400:225 [ 1293.583928] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1293.584070] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1293.589950] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1298.031383] Lustre: server umount lustre-OST0000 complete [ 1299.272155] Lustre: server umount lustre-MDT0000 complete [ 1302.262137] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 1302.738997] Key type lgssc unregistered [ 1302.807744] LNet: 11533:0:(lib-ptl.c:990:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1302.811620] LNet: Removed LNI 192.168.204.111@tcp [ 1316.671977] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1316.709392] alg: No test for adler32 (adler32-zlib) [ 1317.517251] Lustre: Lustre: Build Version: 2.15.61_117_gc7b1037 [ 1317.563519] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 1317.565050] LNet: Accept secure, port 988 [ 1319.091359] Key type lgssc registered [ 1319.286648] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1321.244833] vdc: vdc1 vdc9 [ 1323.574451] vde: vde1 vde9 [ 1325.976748] vdf: vdf1 vdf9 [ 1328.177905] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1329.316208] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1329.339448] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1329.374354] Lustre: lustre-MDT0000: new disk, initializing [ 1329.441961] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1329.458403] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1330.269541] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1332.592084] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1334.077294] Lustre: lustre-OST0000: new disk, initializing [ 1334.078816] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1334.080948] Lustre: Skipped 1 previous similar message [ 1334.102668] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1335.446928] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1337.781917] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1338.264511] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1338.267846] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1338.294752] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1347.889749] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 1347.895101] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1347.903934] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1351.121376] Lustre: server umount lustre-OST0000 complete [ 1352.393975] Lustre: server umount lustre-MDT0000 complete [ 1355.622446] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 1356.131891] Key type lgssc unregistered [ 1356.195907] LNet: 21406:0:(lib-ptl.c:990:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1356.198719] LNet: Removed LNI 192.168.204.111@tcp [ 1364.252159] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1364.282248] alg: No test for adler32 (adler32-zlib) [ 1365.087606] Lustre: Lustre: Build Version: 2.15.61_117_gc7b1037 [ 1365.133362] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 1365.135511] LNet: Accept secure, port 988 [ 1366.663492] Key type lgssc registered [ 1366.874292] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1368.139124] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1369.297539] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1370.069209] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1371.120337] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1372.402887] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1373.658735] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1374.726019] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1379.402274] Lustre: lustre-OST0000: deleting orphan objects from 0x240000400:3 to 0x240000400:33 [ 1382.254467] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 1382.261474] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1382.266263] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 1382.269069] LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 1382.275699] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.204.111@tcp (at 0@lo) [ 1382.760320] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 1382.762755] Lustre: Skipped 1 previous similar message [ 1382.767182] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1382.771266] Lustre: Skipped 1 previous similar message [ 1382.772533] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 1382.775098] Lustre: Skipped 1 previous similar message [ 1382.776780] LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 1382.780009] LustreError: Skipped 1 previous similar message [ 1382.781608] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.204.111@tcp (at 0@lo) [ 1382.784431] Lustre: Skipped 1 previous similar message [ 1383.812540] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 1383.815247] Lustre: Skipped 3 previous similar messages [ 1383.821065] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1383.826639] Lustre: Skipped 3 previous similar messages [ 1383.828774] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 1383.831697] Lustre: Skipped 3 previous similar messages [ 1383.833594] LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 1383.838523] LustreError: Skipped 3 previous similar messages [ 1383.841311] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.204.111@tcp (at 0@lo) [ 1383.843881] Lustre: Skipped 3 previous similar messages [ 1384.974104] LustreError: 23264:0:(osp_precreate.c:670:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 1384.977817] LustreError: 23264:0:(osp_precreate.c:1374:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 1386.023110] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 1386.025089] Lustre: Skipped 7 previous similar messages [ 1386.029667] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1386.033691] Lustre: Skipped 7 previous similar messages [ 1386.035444] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 1386.037461] Lustre: Skipped 7 previous similar messages [ 1386.038700] LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 1386.042639] LustreError: Skipped 7 previous similar messages [ 1386.044552] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.204.111@tcp (at 0@lo) [ 1386.048376] Lustre: Skipped 7 previous similar messages [ 1390.195990] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 1390.198638] Lustre: Skipped 15 previous similar messages [ 1390.204004] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1390.208112] Lustre: Skipped 15 previous similar messages [ 1390.209920] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 1390.212730] Lustre: Skipped 15 previous similar messages [ 1390.214540] LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 1390.218009] LustreError: Skipped 15 previous similar messages [ 1390.219263] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.204.111@tcp (at 0@lo) [ 1390.221719] Lustre: Skipped 15 previous similar messages [ 1398.223663] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 1398.227131] Lustre: Skipped 31 previous similar messages [ 1398.233075] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1398.237756] Lustre: Skipped 31 previous similar messages [ 1398.239538] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 1398.241907] Lustre: Skipped 31 previous similar messages [ 1398.243433] LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 1398.246977] LustreError: Skipped 31 previous similar messages [ 1398.248896] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.204.111@tcp (at 0@lo) [ 1398.253406] Lustre: Skipped 31 previous similar messages [ 1401.757375] LustreError: 23264:0:(osp_precreate.c:670:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 1401.761450] LustreError: 23264:0:(osp_precreate.c:1374:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 1414.262517] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 1414.264638] Lustre: Skipped 68 previous similar messages [ 1414.269506] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1414.273252] Lustre: Skipped 68 previous similar messages [ 1414.274800] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 1414.276524] Lustre: Skipped 68 previous similar messages [ 1414.277786] LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 1414.280261] LustreError: Skipped 68 previous similar messages [ 1414.281634] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.204.111@tcp (at 0@lo) [ 1414.283355] Lustre: Skipped 68 previous similar messages [ 1446.481017] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 1446.483943] Lustre: Skipped 130 previous similar messages [ 1446.488919] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1446.492522] Lustre: Skipped 130 previous similar messages [ 1446.493747] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 1446.496059] Lustre: Skipped 130 previous similar messages [ 1446.496988] LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 1446.502024] LustreError: Skipped 130 previous similar messages [ 1446.504289] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.204.111@tcp (at 0@lo) [ 1446.507672] Lustre: Skipped 130 previous similar messages [ 1487.556644] Lustre: 23227:0:(mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno req@ffff88009acfca80 x1792840817351616/t8589941923(0) o35->d2f7a42d-6753-493a-8620-da8e17ac0a69@192.168.204.11@tcp:542/0 lens 392/456 e 0 to 0 dl 1709786377 ref 1 fl Interpret:/202/0 rc 0/0 job:'' uid:0 gid:0 [ 1496.413361] LustreError: 23264:0:(osp_precreate.c:670:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 1496.416562] LustreError: 23264:0:(osp_precreate.c:1374:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 1510.696970] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 1510.698773] Lustre: Skipped 251 previous similar messages [ 1510.703830] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1510.707752] Lustre: Skipped 251 previous similar messages [ 1510.709763] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 1510.712481] Lustre: Skipped 252 previous similar messages [ 1510.714696] LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 1510.718927] LustreError: Skipped 251 previous similar messages [ 1510.720532] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.204.111@tcp (at 0@lo) [ 1510.723201] Lustre: Skipped 251 previous similar messages [ 1512.165777] LustreError: 23264:0:(osp_precreate.c:670:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 1512.168741] LustreError: 23264:0:(osp_precreate.c:1374:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 1524.380593] LustreError: 23264:0:(osp_precreate.c:670:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 1524.384205] LustreError: 23264:0:(osp_precreate.c:1374:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 1539.664070] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1541.186377] Lustre: server umount lustre-OST0000 complete [ 1542.436107] Lustre: server umount lustre-MDT0000 complete [ 1545.183613] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 1545.624050] Key type lgssc unregistered [ 1545.696483] LNet: 3899:0:(lib-ptl.c:990:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1545.698516] LNet: Removed LNI 192.168.204.111@tcp [ 1553.647799] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1553.675623] alg: No test for adler32 (adler32-zlib) [ 1554.500156] Lustre: Lustre: Build Version: 2.15.61_117_gc7b1037 [ 1554.555747] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 1554.557410] LNet: Accept secure, port 988 [ 1556.085674] Key type lgssc registered [ 1556.332778] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1557.916126] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1559.171593] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1560.139700] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1561.301259] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1562.803982] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1564.148602] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1565.294839] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1569.803323] Lustre: lustre-OST0000: deleting orphan objects from 0x240000400:5395 to 0x240000400:5441 [ 1572.451496] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 1572.972432] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1572.977309] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 1572.980312] LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 1572.984833] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.204.111@tcp (at 0@lo) [ 1572.984868] LustreError: 5763:0:(osp_precreate.c:1282:osp_precreate_thread()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 1574.815960] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1574.821025] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1575.559367] Lustre: server umount lustre-OST0000 complete [ 1593.084404] LustreError: 5763:0:(osp_precreate.c:1282:osp_precreate_thread()) cfs_fail_timeout id 2107 awake [ 1602.545639] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1603.799889] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1604.853936] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1614.538561] LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 1614.542412] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.204.111@tcp (at 0@lo) [ 1617.399153] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 12 sec [ 1618.002638] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff88013730b000.ost_server_uuid 50 [ 1618.368310] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88013730b000.ost_server_uuid in FULL state after 0 sec [ 1619.551998] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1619.558139] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1624.560301] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1625.108909] Lustre: server umount lustre-OST0000 complete [ 1626.353728] Lustre: server umount lustre-MDT0000 complete [ 1629.404052] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 1629.932593] Key type lgssc unregistered [ 1630.014595] LNet: 10632:0:(lib-ptl.c:990:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1630.016480] LNet: Removed LNI 192.168.204.111@tcp [ 1639.431111] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1639.474198] alg: No test for adler32 (adler32-zlib) [ 1640.282647] Lustre: Lustre: Build Version: 2.15.61_117_gc7b1037 [ 1640.326061] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 1640.327837] LNet: Accept secure, port 988 [ 1641.856614] Key type lgssc registered [ 1642.052894] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1643.563644] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1644.721753] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1645.475974] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1647.327250] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1648.701503] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1650.971563] Lustre: lustre-OST0001: new disk, initializing [ 1650.973729] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 1651.002040] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 1652.468562] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1654.094761] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 1654.098724] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 1654.128620] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 1655.078107] Lustre: lustre-OST0000: deleting orphan objects from 0x240000400:5443 to 0x240000400:5473 [ 1656.780901] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1662.140630] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 1677.278941] Lustre: server umount lustre-MDT0000 complete [ 1678.139661] LustreError: 12777:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709786557 with bad export cookie 12912669435805862514 [ 1678.147215] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1688.139367] Lustre: 17482:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1709786557/real 1709786557] req@ffff880095f20700 x1792841109135680/t0(0) o39->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1709786567 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 [ 1688.166387] Lustre: server umount lustre-OST0000 complete [ 1699.196413] Lustre: 17710:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1709786568/real 1709786568] req@ffff88012fe50e00 x1792841109136128/t0(0) o39->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1709786578 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 [ 1699.281058] Lustre: server umount lustre-OST0001 complete [ 1706.437288] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 1708.905148] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1709.978529] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1714.001509] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1716.645493] Lustre: mylustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 1716.648115] Lustre: Skipped 1 previous similar message [ 1718.643122] Lustre: mylustre-OST0000: deleting orphan objects from 0x240000400:5476 to 0x240000400:5505 [ 1719.074015] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1722.540707] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1723.983994] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 1733.651269] Lustre: mylustre-MDT0000-lwp-OST0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1733.655929] Lustre: Skipped 1 previous similar message [ 1733.660917] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1733.663455] Lustre: Skipped 1 previous similar message [ 1738.656447] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1738.658999] Lustre: Skipped 1 previous similar message [ 1739.117308] Lustre: server umount mylustre-MDT0000 complete [ 1740.062113] LustreError: 23531:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709786619 with bad export cookie 12912669435805864677 [ 1740.064559] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1740.077506] LustreError: 23531:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1740.086746] Lustre: server umount mylustre-OST0000 complete [ 1741.064408] Lustre: server umount mylustre-OST0001 complete [ 1751.875525] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 1755.061662] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1756.802606] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1763.918111] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1764.603315] LustreError: 137-5: tfs-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1764.613735] Lustre: tfs-OST0000: deleting orphan objects from 0x240000400:5476 to 0x240000400:5537 [ 1768.640049] Lustre: tfs-OST0001: deleting orphan objects from 0x280000400:35 to 0x280000400:65 [ 1768.951120] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1772.220339] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1773.582746] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 1782.159971] Lustre: tfs-MDT0000-lwp-OST0000: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1782.165010] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 1782.169170] Lustre: Skipped 1 previous similar message [ 1786.751240] Lustre: server umount tfs-MDT0000 complete [ 1787.873164] LustreError: 27327:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709786666 with bad export cookie 12912669435805865888 [ 1787.877391] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1787.881382] LustreError: 27327:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1787.902910] Lustre: server umount tfs-OST0000 complete [ 1789.187084] Lustre: server umount tfs-OST0001 complete [ 1803.738160] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 1807.482669] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1807.487102] Lustre: Skipped 2 previous similar messages [ 1809.225323] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1814.724806] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1817.259542] LustreError: 137-5: lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1817.286742] Lustre: lustre-OST0000: deleting orphan objects from 0x240000400:5476 to 0x240000400:5569 [ 1820.041912] Lustre: lustre-OST0001: deleting orphan objects from 0x280000400:35 to 0x280000400:97 [ 1820.792065] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1825.372966] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1827.116449] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 1837.858611] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 23:45:16 (1709786716) [ 1853.202097] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1853.204569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1853.222656] Lustre: Skipped 2 previous similar messages [ 1856.306158] Lustre: server umount lustre-MDT0000 complete [ 1858.866424] LustreError: 4900:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709786737 with bad export cookie 12912669435805867127 [ 1858.870532] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1864.898692] Lustre: server umount lustre-OST0000 complete [ 1869.196077] Lustre: server umount lustre-OST0001 complete [ 1871.701040] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_hostid [ 1873.740299] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 1875.566244] hrtimer: interrupt took 2651110 ns [ 1877.365130] vdc: vdc1 vdc9 [ 1882.700645] vde: vde1 vde9 [ 1888.085607] vdf: vdf1 vdf9 [ 1891.927306] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1891.955944] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1891.995010] Lustre: lustre-MDT0000: new disk, initializing [ 1892.107537] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1892.110439] Lustre: Skipped 2 previous similar messages [ 1892.131635] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1893.136947] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1897.424681] Lustre: lustre-OST0000: new disk, initializing [ 1897.427241] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1897.430135] Lustre: Skipped 1 previous similar message [ 1898.630487] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1898.638733] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1898.809126] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1899.380319] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1904.218413] Lustre: lustre-OST0001: new disk, initializing [ 1904.220763] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 1905.270498] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 1905.274263] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 1905.329589] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 1906.171885] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1914.418806] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 23:46:33 (1709786793) [ 1921.495804] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 23:46:40 (1709786800) [ 1933.635226] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1933.638715] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1933.643782] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1933.646539] Lustre: Skipped 1 previous similar message [ 1935.085308] Lustre: server umount lustre-OST0000 complete [ 1935.281017] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1939.379228] Lustre: server umount lustre-MDT0000 complete [ 1944.782476] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 1947.983149] LustreError: 9146:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709786826 with bad export cookie 12912669435805868247 [ 1947.988057] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1957.994403] Lustre: 16308:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1709786826/real 1709786826] req@ffff88009b670700 x1792841109184512/t0(0) o39->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1709786836 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 [ 1964.069805] Lustre: server umount lustre-OST0001 complete [ 1965.793208] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_hostid [ 1970.662804] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 1973.600104] vdc: vdc1 vdc9 [ 1976.788001] vde: vde1 vde9 [ 1981.259698] vdf: vdf1 vdf9 [ 1986.257798] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1986.318328] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1986.406140] Lustre: lustre-MDT0000: new disk, initializing [ 1986.653888] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1986.656008] Lustre: Skipped 2 previous similar messages [ 1986.691733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1988.665271] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1992.543161] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1996.079870] Lustre: lustre-OST0000: new disk, initializing [ 1996.090347] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1996.092094] Lustre: Skipped 1 previous similar message [ 1997.802170] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1997.807217] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1997.932634] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1998.857921] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2002.975803] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2007.729476] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2007.742437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2011.785904] Lustre: server umount lustre-OST0000 complete [ 2014.442954] Lustre: server umount lustre-MDT0000 complete [ 2020.131537] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 2021.136461] Key type lgssc unregistered [ 2021.265118] LNet: 23551:0:(lib-ptl.c:990:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2021.269863] LNet: Removed LNI 192.168.204.111@tcp [ 2038.601333] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2038.718086] alg: No test for adler32 (adler32-zlib) [ 2039.699483] Lustre: Lustre: Build Version: 2.15.61_117_gc7b1037 [ 2039.853895] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 2039.858010] LNet: Accept secure, port 988 [ 2041.438613] Key type lgssc registered [ 2042.102173] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2045.732053] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2047.391479] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2049.460289] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2051.945299] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2055.216707] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2057.210577] Lustre: lustre-OST0000: deleting orphan objects from 0x240000400:3 to 0x240000400:33 [ 2058.181768] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2060.287219] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2060.692965] Lustre: Setting parameter lustre-OST0000.ost.unknown_param in log lustre-OST0000 [ 2062.195181] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2062.205027] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2067.201397] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2067.800418] Lustre: server umount lustre-OST0000 complete [ 2070.309664] Lustre: server umount lustre-MDT0000 complete [ 2075.322345] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 2076.651239] Key type lgssc unregistered [ 2076.816882] LNet: 28160:0:(lib-ptl.c:990:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2076.830037] LNet: Removed LNI 192.168.204.111@tcp [ 2085.157431] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2085.192598] alg: No test for adler32 (adler32-zlib) [ 2086.104465] Lustre: Lustre: Build Version: 2.15.61_117_gc7b1037 [ 2086.206763] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 2086.210851] LNet: Accept secure, port 988 [ 2087.827435] Key type lgssc registered [ 2088.660676] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2093.262077] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2110.511511] LustreError: 29300:0:(mgc_request.c:251:do_config_log_add()) MGC192.168.204.111@tcp: failed processing log, type 1: rc = -5 [ 2146.535557] LustreError: 29300:0:(mgc_request.c:251:do_config_log_add()) MGC192.168.204.111@tcp: failed processing log, type 4: rc = -110 [ 2296.612388] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2299.294936] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2301.459888] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2302.715590] Lustre: server umount lustre-OST0000 complete [ 2308.097034] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 2309.576742] Key type lgssc unregistered [ 2309.799973] LNet: 30980:0:(lib-ptl.c:990:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2309.806034] LNet: Removed LNI 192.168.204.111@tcp [ 2332.673011] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2332.724041] alg: No test for adler32 (adler32-zlib) [ 2333.794605] Lustre: Lustre: Build Version: 2.15.61_117_gc7b1037 [ 2333.941891] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 2333.946660] LNet: Accept secure, port 988 [ 2335.527581] Key type lgssc registered [ 2336.270849] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2366.547898] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2366.800714] Lustre: 5017:0:(osd_scrub.c:915:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 2366.805107] Lustre: 5017:0:(osd_scrub.c:915:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0xd:0x0]: rc = 0 [ 2366.809697] Lustre: 5017:0:(osd_scrub.c:915:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x2:0x0]: rc = 0 [ 2366.813318] Lustre: 5017:0:(osd_scrub.c:915:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0x9:0x0]: rc = 0 [ 2366.817271] Lustre: 5017:0:(osd_scrub.c:915:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xb:0x0]: rc = 0 [ 2366.822292] Lustre: 5017:0:(osd_scrub.c:915:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 2366.866549] Lustre: 5017:0:(osd_scrub.c:915:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xa:0x0]: rc = 0 [ 2366.870769] Lustre: 5017:0:(osd_scrub.c:915:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 2366.874755] Lustre: 5017:0:(osd_scrub.c:915:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 2366.878149] Lustre: 5017:0:(osd_scrub.c:915:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 2366.886272] Lustre: 5017:0:(osd_scrub.c:915:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 2366.893670] Lustre: 5017:0:(osd_scrub.c:915:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 2366.896965] Lustre: 5017:0:(osd_scrub.c:915:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 2366.900050] Lustre: 5017:0:(osd_scrub.c:915:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 2366.909361] Lustre: 5017:0:(osd_scrub.c:915:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 2371.305177] Lustre: server umount MGS complete [ 2372.317014] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2372.373774] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2372.528912] Lustre: lustre-MDT0000: Aborting client recovery [ 2373.505486] Lustre: 6413:0:(osd_scrub.c:915:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 2373.514430] Lustre: 6413:0:(osd_scrub.c:915:osd_index_restore()) lustre-MDT0001: restore index 'nodemap' with [0x200000003:0x7:0x0]: rc = 0 [ 2373.518868] Lustre: 6413:0:(osd_scrub.c:915:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x4:0x0]: rc = 0 [ 2373.525435] Lustre: 6413:0:(osd_scrub.c:915:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x5:0x0]: rc = 0 [ 2373.532214] Lustre: 6413:0:(osd_scrub.c:915:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 2373.538625] Lustre: 6413:0:(osd_scrub.c:915:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x3:0x0]: rc = 0 [ 2373.543847] Lustre: 6413:0:(osd_scrub.c:915:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 2373.549925] Lustre: 6413:0:(osd_scrub.c:915:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 2373.561251] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2373.751943] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2373.837826] Lustre: lustre-MDT0001: Aborting client recovery [ 2375.118048] Lustre: 7066:0:(osd_scrub.c:915:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 2375.122484] Lustre: 7066:0:(osd_scrub.c:915:osd_index_restore()) lustre-OST0000: restore index 'nodemap' with [0x200000003:0x2:0x0]: rc = 0 [ 2375.127662] Lustre: 7066:0:(osd_scrub.c:915:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x5:0x0]: rc = 0 [ 2375.132061] Lustre: 7066:0:(osd_scrub.c:915:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x4:0x0]: rc = 0 [ 2375.137028] Lustre: 7066:0:(osd_scrub.c:915:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x6:0x0]: rc = 0 [ 2375.141142] Lustre: 7066:0:(osd_scrub.c:915:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 2375.147144] Lustre: 7066:0:(osd_scrub.c:915:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 2375.157105] Lustre: 7066:0:(osd_scrub.c:915:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 2375.270433] Lustre: Found index 0 for lustre-OST0000, updating log [ 2375.439831] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2375.499973] Lustre: lustre-OST0000: Aborting recovery [ 2376.824302] Lustre: 7745:0:(osd_scrub.c:915:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 2376.828692] Lustre: 7745:0:(osd_scrub.c:915:osd_index_restore()) lustre-OST0001: restore index 'nodemap' with [0x200000003:0x2:0x0]: rc = 0 [ 2376.832998] Lustre: 7745:0:(osd_scrub.c:915:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 2376.837770] Lustre: 7745:0:(osd_scrub.c:915:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x5:0x0]: rc = 0 [ 2376.842768] Lustre: 7745:0:(osd_scrub.c:915:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x4:0x0]: rc = 0 [ 2376.847423] Lustre: 7745:0:(osd_scrub.c:915:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x6:0x0]: rc = 0 [ 2376.852444] Lustre: 7745:0:(osd_scrub.c:915:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 2376.865664] Lustre: 7745:0:(osd_scrub.c:915:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 2376.887671] Lustre: Found index 1 for lustre-OST0001, updating log [ 2376.919345] LustreError: 8159:0:(osd_index.c:217:__osd_xattr_load_by_oid()) lustre-OST0001: can't get bonus, rc = -2 [ 2377.153589] Lustre: lustre-OST0001: Aborting recovery [ 2382.110946] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6 to 0x0:33 [ 2382.111837] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6 to 0x0:33 [ 2382.158873] LustreError: 6253:0:(lod_dev.c:525:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 10, retries 0, failed: rc = -108 [ 2382.198821] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000bd0 [ 2382.220788] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000bd0 [ 2383.171462] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:5 to 0x280000400:33 [ 2383.182441] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:5 to 0x2c0000400:33 [ 2385.891830] LustreError: 6073:0:(mdt_identity.c:112:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 1838us: rc = -2 [ 2386.404509] LustreError: 6072:0:(mdt_identity.c:112:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 952us: rc = -2 [ 2386.415601] LustreError: 6072:0:(mdt_identity.c:112:mdt_identity_do_upcall()) Skipped 37 previous similar messages [ 2398.144085] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2398.145251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2398.165526] Lustre: Skipped 3 previous similar messages [ 2403.152402] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2403.154603] Lustre: Skipped 6 previous similar messages [ 2405.425631] Lustre: server umount lustre-MDT0000 complete [ 2406.976826] LustreError: 6056:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709787285 with bad export cookie 5478406519362280649 [ 2406.981491] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2408.160448] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2408.161518] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2408.168421] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2408.175283] LustreError: Skipped 1 previous similar message [ 2409.407124] Lustre: server umount lustre-MDT0001 complete [ 2412.684939] Lustre: server umount lustre-OST0000 complete [ 2420.060472] Lustre: server umount lustre-OST0001 complete [ 2429.566286] Lustre: DEBUG MARKER: == conf-sanity test 108b: migrate from ZFS to ldiskfs ==== 23:55:08 (1709787308) [ 2430.413414] Lustre: DEBUG MARKER: SKIP: conf-sanity test_108b ldiskfs only test [ 2435.373721] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 23:55:13 (1709787313) [ 2436.319732] Lustre: DEBUG MARKER: SKIP: conf-sanity test_109a LU-8727: no implementation for ZFS [ 2440.905015] Lustre: DEBUG MARKER: == conf-sanity test 109b: test lctl clear_conf one config ========================================================== 23:55:19 (1709787319) [ 2441.910554] Lustre: DEBUG MARKER: SKIP: conf-sanity test_109b LU-8727: no implementation for ZFS [ 2444.067807] Lustre: DEBUG MARKER: SKIP: conf-sanity test_110 skipping ALWAYS excluded test 110 [ 2445.078015] Lustre: DEBUG MARKER: SKIP: conf-sanity test_111 skipping SLOW test 111 [ 2448.275419] Lustre: DEBUG MARKER: == conf-sanity test 112a: mount OST with no_create option ========================================================== 23:55:26 (1709787326) [ 2451.898748] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2451.901519] Lustre: Skipped 1 previous similar message [ 2454.122106] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2456.706173] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2460.047445] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2462.019887] Lustre: lustre-OST0000: deleting orphan objects from 0x240000400:3 to 0x240000400:65 [ 2463.245780] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2466.006044] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2469.927587] Lustre: lustre-OST0001: new disk, initializing [ 2469.933438] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 2469.987026] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2472.810096] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2477.566260] LustreError: 16026:0:(osp_precreate.c:988:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 2477.956096] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2478.065102] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2478.582664] LustreError: 16026:0:(osp_precreate.c:988:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 2479.602525] LustreError: 16026:0:(osp_precreate.c:988:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 2479.677303] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800a851c800.ost_server_uuid 50 [ 2480.542378] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800a851c800.ost_server_uuid in FULL state after 0 sec [ 2481.610633] LustreError: 16026:0:(osp_precreate.c:988:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 2481.613473] LustreError: 16026:0:(osp_precreate.c:988:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [ 2482.341408] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800a851c800.ost_server_uuid 50 [ 2483.352245] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8800a851c800.ost_server_uuid in FULL state after 0 sec [ 2483.713463] Lustre: lustre-MDT0000: Client 59effffe-8008-4aaf-8fa8-78545c87ac20 (at 192.168.204.11@tcp) reconnecting [ 2485.644612] LustreError: 16026:0:(osp_precreate.c:988:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 2485.653898] LustreError: 16026:0:(osp_precreate.c:988:osp_precreate_cleanup_orphans()) Skipped 3 previous similar messages [ 2493.710686] LustreError: 16026:0:(osp_precreate.c:988:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 2493.713426] LustreError: 16026:0:(osp_precreate.c:988:osp_precreate_cleanup_orphans()) Skipped 7 previous similar messages [ 2495.825521] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 2495.829951] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 2495.937255] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 2507.509535] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2507.527972] Lustre: Skipped 1 previous similar message [ 2507.538459] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2507.545488] Lustre: Skipped 1 previous similar message [ 2512.513297] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2512.520246] Lustre: Skipped 1 previous similar message [ 2513.180594] Lustre: server umount lustre-OST0001 complete [ 2513.803483] LustreError: 137-5: lustre-OST0001: not available for connect from 192.168.204.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2517.121241] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2517.126580] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2517.531875] LustreError: 137-5: lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2522.548450] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2522.551260] Lustre: Skipped 1 previous similar message [ 2522.553549] LustreError: 137-5: lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2522.837798] Lustre: server umount lustre-OST0000 complete [ 2525.293291] Lustre: server umount lustre-MDT0000 complete [ 2531.867434] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 2533.964841] Key type lgssc unregistered [ 2534.093427] LNet: 18131:0:(lib-ptl.c:990:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2534.101721] LNet: Removed LNI 192.168.204.111@tcp [ 2571.258008] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2571.317832] alg: No test for adler32 (adler32-zlib) [ 2572.218222] Lustre: Lustre: Build Version: 2.15.61_117_gc7b1037 [ 2572.374341] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 2572.380793] LNet: Accept secure, port 988 [ 2573.966084] Key type lgssc registered [ 2574.656513] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2579.326980] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2580.941522] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2583.331816] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2588.503090] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2590.488210] LustreError: 137-5: lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2590.501329] Lustre: lustre-OST0000: deleting orphan objects from 0x240000400:68 to 0x240000400:97 [ 2591.682207] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2595.489798] LustreError: 137-5: lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2596.289423] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2599.637338] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2601.256449] Lustre: lustre-OST0001: deleting orphan objects from 0x280000400:3 to 0x280000400:33 [ 2604.805779] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2608.885435] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 2621.285595] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2621.317220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2621.326108] Lustre: Skipped 1 previous similar message [ 2624.218241] Lustre: server umount lustre-MDT0000 complete [ 2627.461266] LustreError: 21347:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709787506 with bad export cookie 3449272389218356635 [ 2627.466434] LustreError: 21347:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2627.470253] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2627.499951] Lustre: server umount lustre-OST0000 complete [ 2630.178675] Lustre: server umount lustre-OST0001 complete [ 2638.937665] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 2644.540980] Lustre: lustre-MDT0000: set dev_rdonly on this device [ 2644.920959] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2646.980294] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2651.317390] Lustre: lustre-OST0000: set dev_rdonly on this device [ 2651.502947] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2654.198363] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2654.505761] LustreError: 137-5: lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2659.300647] Lustre: lustre-OST0001: set dev_rdonly on this device [ 2659.506609] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2662.242279] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2674.502411] Lustre: server umount lustre-MDT0000 complete [ 2677.128445] LustreError: 29623:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709787556 with bad export cookie 3449272389218357867 [ 2677.134075] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2677.140537] LustreError: 29623:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2677.157473] Lustre: server umount lustre-OST0000 complete [ 2679.959911] Lustre: server umount lustre-OST0001 complete [ 2686.911104] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 2691.346504] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2693.407954] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2700.751362] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2701.884870] LustreError: 137-5: lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2701.896050] Lustre: lustre-OST0000: deleting orphan objects from 0x240000400:68 to 0x240000400:129 [ 2705.614648] Lustre: lustre-OST0001: deleting orphan objects from 0x280000400:3 to 0x280000400:65 [ 2707.349978] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2711.485501] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2713.333995] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 2724.368356] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2724.370487] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2724.374752] Lustre: Skipped 2 previous similar messages [ 2726.021073] Lustre: server umount lustre-MDT0000 complete [ 2727.931487] LustreError: 31469:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709787606 with bad export cookie 3449272389218358462 [ 2727.933310] LustreError: 166-1: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2727.944786] LustreError: 31469:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2727.961857] Lustre: server umount lustre-OST0000 complete [ 2729.970366] Lustre: server umount lustre-OST0001 complete [ 2733.442563] Lustre: DEBUG MARKER: SKIP: conf-sanity test_114 skipping SLOW test 114 [ 2734.445032] Lustre: DEBUG MARKER: SKIP: conf-sanity test_115 skipping excluded test 115 [ 2737.369662] Lustre: DEBUG MARKER: == conf-sanity test 116: big size MDT support ============ 00:00:16 (1709787616) [ 2738.089725] Lustre: DEBUG MARKER: SKIP: conf-sanity test_116 ldiskfs only test [ 2742.032977] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 00:00:20 (1709787620) [ 2745.238557] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2745.241387] Lustre: Skipped 2 previous similar messages [ 2746.949654] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2749.368850] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2755.478249] LustreError: 137-5: lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2755.488044] Lustre: lustre-OST0000: deleting orphan objects from 0x240000400:68 to 0x240000400:161 [ 2755.543910] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2757.947839] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2764.313425] Lustre: server umount lustre-OST0000 complete [ 2776.941705] Lustre: 6949:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1709787645/real 1709787645] req@ffff8800959e1180 x1792842086256960/t0(0) o9->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1709787655 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 [ 2777.186539] Lustre: server umount lustre-MDT0000 complete [ 2781.905040] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 2783.010091] Key type lgssc unregistered [ 2783.152913] LNet: 7555:0:(lib-ptl.c:990:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2783.157101] LNet: Removed LNI 192.168.204.111@tcp [ 2813.800544] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2813.972099] alg: No test for adler32 (adler32-zlib) [ 2814.904904] Lustre: Lustre: Build Version: 2.15.61_117_gc7b1037 [ 2815.049223] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 2815.053676] LNet: Accept secure, port 988 [ 2816.707343] Key type lgssc registered [ 2817.320929] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2820.311394] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2821.732785] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2823.239209] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2825.475067] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2826.677695] Lustre: Failing over lustre-MDT0000 [ 2826.959796] Lustre: server umount lustre-MDT0000 complete [ 2841.809549] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2843.679755] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2847.371832] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 2849.901135] Lustre: server umount lustre-MDT0000 complete [ 2853.727419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2855.216672] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2857.321165] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2858.238989] Lustre: Failing over lustre-MDT0000 [ 2858.394315] Lustre: server umount lustre-MDT0000 complete [ 2872.357475] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2874.041063] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2877.993818] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 2880.859833] Lustre: server umount lustre-MDT0000 complete [ 2889.306327] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 00:02:47 (1709787767) [ 2890.605102] Lustre: DEBUG MARKER: SKIP: conf-sanity test_122a needs >= 2 MDTs [ 2895.494896] Lustre: DEBUG MARKER: == conf-sanity test 123aa: llog_print works with FIDs and simple names ========================================================== 00:02:54 (1709787774) [ 2898.645928] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2900.549305] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2903.353991] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2906.976174] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2908.963781] LustreError: 137-5: lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2908.989685] Lustre: lustre-OST0000: deleting orphan objects from 0x240000400:68 to 0x240000400:193 [ 2910.110257] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2912.467209] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2912.819942] Lustre: MGS: Client a76d2fe4-57ca-420d-bad8-d5df07dd8bbd (at 192.168.204.11@tcp) reconnecting [ 2913.968923] LustreError: 137-5: lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2914.095633] Lustre: DEBUG MARKER: conf-sanity test_123aa: @@@@@@ FAIL: client start failed [ 2918.977137] LustreError: 137-5: lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2923.986034] LustreError: 137-5: lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2925.980796] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 00:03:24 (1709787804) [ 2928.677097] Lustre: DEBUG MARKER: conf-sanity test_123ab: @@@@@@ FAIL: MDT start failed [ 2928.996420] LustreError: 137-5: lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.